Working with Hadoop with two datasets -

-

i'm working hadoop , try make interection function 2 datasets what's best scenario. can load 1 memory , intersect in map function other dataset if dataset large ram memory not solution, thoughts?

thanks answers, i'll try out these see what's best solution.

it's tough maneuver intersection in mapred compared other family of platforms around hadoop api. mentioned hive (super easy intersections if have sql background), may consider:

  • pig
  • cascading (specifically cogroup if memory concern , hashjoin if isn't)

Popular posts from this blog

Php - Delimiter must not be alphanumeric or backslash -

-
this question has answer here: delimiter must not alphanumeric or backslash , preg_match 5 answers this question has been asked don't know how fix problem in situation. getting following php errror: php message: php warning: preg_match(): delimiter must not alphanumeric or backslash in... on line 227 line 227 following: if (preg_match($match, $_server['http_user_agent'])) and function: $oslist = array ( 'windows 3.11' => 'win16', 'windows 95' => '(windows 95)|(win95)|(windows_95)', 'windows 98' => '(windows 98)|(win98)', 'windows 2000' => '(windows nt 5.0)|(windows 2000)', 'windows xp' => '(windows nt 5.1)|(windows xp)', 'windows server 2003' => '(windows nt 5.2)', 'windows vista' => '(windows nt 6.0)', 'w...
Read more

c# - How to change the "Applies To" field under folder auditing options programatically (.NET) -

-
Image
i trying set "applies to" field under folder auditing options programatically. in msdn, the code example there uses filesystemauditrule class add new audit rule folder. there nothing obvious in class set particular audit rule needs applied to. this code using set permissions: const string myfolder = @"s:\temp\somefoldertoaudit"; var account = new securityidentifier(wellknownsidtype.worldsid, null).translate(typeof(ntaccount)); filesecurity fsecurity = file.getaccesscontrol(myfolder, accesscontrolsections.audit); fsecurity.addauditrule(new filesystemauditrule(account, filesystemrights.writedata | filesystemrights.delete | filesystemrights.changepermissions, auditflags.success)); file.setaccesscontrol(myfolder, fsecurity); this creates audit rules nicely except highlighted option below: i need " this folder, subfolders , files " example or other " this folder only ". don't want traverse directories , files , set same auditing r...
Read more

c++ - Ambiguity when using boost::assign::list_of to construct a std::vector -

-
this code: std::vector<int>(boost::assign::list_of<int>(1)(2)(3)); gives error: main.cpp: in member function 'void <unnamed>::requesthandler::processrequest(foo&, bar, unsigned int, unsigned int*, const char*, boost::shared_ptr<baz::ioutput>&)': main.cpp:450: error: call of overloaded 'vector(boost::assign_detail::generic_list<int>&)' ambiguous /4.4.2/bits/stl_vector.h:241: note: candidates are: std::vector<_tp, _alloc>::vector(const std::vector<_tp, _alloc>&) [with _tp = int, _alloc = std::allocator<int>] /4.4.2/bits/stl_vector.h:227: note: std::vector<_tp, _alloc>::vector(size_t, const _tp&, const _alloc&) [with _tp = int, _alloc = std::allocator<int>] /4.4.2/bits/stl_vector.h:215: note: std::vector<_tp, _alloc>::vector(const _alloc&) [with _tp = int, _alloc = std::allocator<int>] when compiled gcc 4.4.2. how can revolve iss...
Read more