[00:00] creationix: mape: no, was the file the actual nStore datafile? [00:00] mape: yeah [00:00] mape: oh thought I mentioned that [00:00] creationix: ok, I'll test it [00:01] creationix: though yours is just one long row, my test if 27k small rows [00:01] creationix: so yours should be even faster [00:01] creationix: (for startup at least) [00:01] jacoblyles has joined the channel [00:01] creationix: I only read and parse the meta per row at startup [00:02] creationix: the data is only read on demand [00:03] creationix: mape: 0m0.096s [00:03] mape: yeah, so no issue [00:03] ryah: ashleydev: i got interrupted [00:03] ryah: but it's not building out of the box [00:03] ashleydev: :( [00:03] creationix: mape: one side effect is I'll have to change the api since startup isn't blocking anymore [00:03] ryah: the fixes i was making were obvious and easy [00:04] mape: creationix: easy enough to fix :) [00:04] ryah: very nice error messages [00:04] creationix: ryah: yes they are [00:04] creationix: almost magical [00:04] d0k: :) [00:04] ashleydev: and all the more so after using gcc ... [00:05] ryah: i would really like to get node building on it [00:05] mape: ryah: So dtrace will be implemented now that you are work buddy with the creator? [00:05] ryah: but i need to sit down for 2 hours and port v8 [00:05] ryah: mape: yes [00:05] ryah: mape: he's my new boss :) [00:05] mape: Met him yet? [00:05] mape: Ah, nice [00:05] ryah: yes [00:05] d0k: I have a (somewhat hackish) patch to build v8 with clang btw [00:06] stepheneb has joined the channel [00:06] mape: He knew about node.js? [00:06] ryah: mape: no, but he likes it [00:06] ryah: he's into js [00:06] softdrink has joined the channel [00:06] mape: cool [00:06] _announcer: Twitter: "@travisbell Ya, it's sweet. I actually had trouble building node.js, so decided to just use brew install node.js, which worked beautifully." -- James Dennes. http://twitter.com/jdennes/status/19866129172 [00:06] ryah: yeah [00:07] creationix: dtrace, that's exciting [00:07] mape: ryah: You had anything to do with the hiring? ;) [00:09] creationix: heh, I found a quick way to grow a test file [00:09] creationix: cat sessions.db >> sessions.db [00:09] creationix: I was expecting it to double the file, but cat seems to pick up the new data [00:09] creationix: now I have a test file with 5.6 million lines [00:10] creationix: and likely some corrupted [00:10] mape: hehe [00:10] _announcer: Twitter: ""You should be urinating right now" @ryah discusses Node.JS, a web server for real-time information streaming http://bit.ly/aPKqOB" -- glennkelman. http://twitter.com/glennkelman/status/19866392866 [00:11] creationix: well, memory usage is staying level [00:11] creationix: that's a good sign [00:11] ryah: wow. [00:12] EyePulp has joined the channel [00:12] creationix: 30% of the cpu time is in JSON.parse [00:12] creationix: well, I assume v8::internal::Runtime_compile [00:13] tpryme: creationix: How are you benchmarking node.js CPU by method invocation? [00:13] creationix: osx process sampler [00:13] tpryme: creationix: I've been trying to find a tool for that [00:13] tpryme: creationix: Anything equivalent in ubuntu/linux? [00:14] WALoeIII has joined the channel [00:14] creationix: ok, finished, 4 minutes to find and parse 5.6 million json objects scattered through the file [00:14] tmpvar: tpryme, i think you can use valgrind [00:14] creationix: I hope I can make it faster [00:15] creationix: maybe I just need to only store the raw key instead of a meta json object [00:15] mjr_: tpryme: you can use oprofile on linux [00:15] tpryme: creationix: mjr_: thanks will look into that [00:15] creationix: I thought JSON.parse was fast? [00:15] mjr_: You can also use the V8 profiler by running node --prof [00:15] tpryme: creationix: What are you trying to make run faster? [00:15] creationix: the data looks like {"key":"somelonguuid"} [00:16] creationix: tpryme: because it sucks if my server takes 4 minutes to startup with only 4 million rows in the table [00:16] tpryme: creationix: using which lib? [00:16] tmpvar: yikes, thats java slow [00:16] creationix: my custom db [00:16] tmpvar: :P [00:16] Apage43: "java slow" [00:16] creationix: nStore [00:16] Apage43: is that like xbox huge [00:17] creationix: I guess since it's non-blocking I can make it startup anyway, and just hold up queries against unknown keys till all the data has been read [00:17] creationix: I'd have to lock writes too probably [00:19] ryah: d0k: are you going to contribute them back to v8 [00:20] d0k: I hope so, but I'm still not sure what the actual issue is [00:20] tpryme: mjr_: Do you know of a good url outlining node --prof? [00:21] _announcer: Twitter: "User-friendly command-line parameters in #nodejs (mostly like Text::Abbreviate for #perl folks) http://j.mp/b4VHMs #snippet" -- Frank Grimm. http://twitter.com/frank_grimm/status/19867057901 [00:22] mjr_: tpryme: I don't. I read through the files in deps/v8 to figure out how it worked. [00:22] mjr_: basically you need to build the "d8" executable, and then you can run linux-tick-processor on the output. [00:22] creationix: ok, if I remove the JSON.parse and cheat (since I KNOW the offset of the key) the next slowest parts are new Buffer at 20% and READ at 24% [00:22] tpryme: creationix: maybe lazy invoke the JSON.parse. keep a table mapping string indices to particular keys and then JSON.parse it upon a key lookup? [00:23] creationix: tpryme: well, I need the key to build the index, it can't be delayed [00:23] creationix: I think I'll just change my format to not allow meta info, and only store the key and offsets in the index [00:23] kodisha has joined the channel [00:23] creationix: alright 2:30, that's better than 4:00 [00:23] Yuffster has joined the channel [00:24] creationix: now new Buffer is killing me [00:24] creationix: I'm using the string version of fs.read() maybe that internally creates a new buffer each time [00:24] creationix: I can limit key size to something sane (1024 chars) and re-use a buffer [00:24] mjr_: new Buffer is expensive at the moment. You gotta re-use them somehow [00:26] creationix: is there no way to read a file without creating buffers [00:26] creationix: I know need the string value [00:26] creationix: I guess that makes sense, the c api requires a buffer [00:27] creationix: s/know/only/ [00:27] tpryme: creationix: You might not need to limit the key size. You can double the size of the re-usable buffer as needed, no? [00:27] creationix: tpryme: true [00:27] creationix: I'll keep a pool of reusable buffers then [00:27] creationix: well, just double and throw away the old one [00:28] satori_ has joined the channel [00:28] creationix: no harm keeping the bigger one around [00:28] creationix: hopefully startup is only a few seconds [00:28] tpryme: creationix: I think new Buffer is what's killing my app, too [00:28] creationix: yeah, seeing these percentages in the sampler is awesome [00:31] tpryme: creationix: Do you know if Buffer.slice invokes new Buffer? [00:31] ryah: tpryme: what do you mean? [00:31] ryah: it doesn't allocate new spce [00:31] creationix: tpryme: it makes a new buffer in js, but shares the old ram [00:31] ryah: but it creates a new js object [00:31] creationix: yeah [00:32] tpryme: creationix: ryah: so basically shouldn't take a hit on your code. right. gotcha [00:32] ryah: tpryme: new objects are kinda expensive [00:32] ryah: but - no [00:32] Aredridel has joined the channel [00:32] creationix: yeah, slice is pretty fast, just be sure to remember it's sharing the same memory as another buffer [00:33] ryah: i'm surprised Buffer is slow enough for mjr to notice it [00:33] ryah: i mean - it is slow - it allocates memory [00:33] ryah: but is the bottleneck actually malloc? [00:33] ryah: that'd be kind of crazy [00:33] ryah: also good [00:33] mjr_: I believe it is malloc, yes. [00:33] Tim_Smart has joined the channel [00:33] ryah: because we can just do a mem pool [00:34] mjr_: oh yes [00:34] ryah: for the "blobs" [00:34] mjr_: And then, dynamically growable buffers [00:34] mjr_: And then, world domination. [00:34] tpryme: mjr_: Yeah that would be good if it was out of the box [00:34] ryah: no - no growing [00:34] ashleydev has joined the channel [00:34] _announcer: Twitter: "You Should Be Urinating Right Now. @ryah talk at @Redfin on Node.JS, a Real-Time Web Server he created using JavaScript http://j.mp/aVaAII" -- Peter Boctor. http://twitter.com/boctor/status/19867932139 [00:34] creationix: I just need a function that can read from a file at a given position and length and return me a utf8 string [00:35] mjr_: I wouldn't tackle growing them for a LONG time. [00:35] creationix: I don't care what buffer pooling magic there is under the hood [00:35] mjr_: Just getting a better allocator I think will win huge. [00:35] mjr_: I'd much rather we fix some other things like aggregating writes at the end of the tick, etc. [00:35] creationix: so creating a new string is faster than creating a new buffer of the same size? [00:35] mjr_: creationix: WAY faster [00:36] mjr_: except for very large strings/buffers [00:36] creationix: good, though in my case it doesn't matter since JSON.parse needs a string anyway [00:36] _announcer: Twitter: "Having some fun with Node.js, Expressjs, and Connect" -- Bradley Price. http://twitter.com/bradleyprice/status/19868029007 [00:36] tpryme: creationix: I would say use redis SUBSTR but you're doing this for nstore [00:36] mjr_: But you can make all kinds of temporary strings, and V8 blazes through them. [00:36] creationix: yeah, these at startup are usually about 60 bytes long [00:37] Tobsn: is here someone into RTP/DSP advertising? [00:38] creationix: I can't ever find things on the api docs [00:38] creationix: where is Buffer's toString? [00:38] creationix: ahh, found it [00:38] tpryme: You can just call toString("utf8") [00:39] ryah: the docs need help [00:40] mjr_: docs, yeah [00:40] mjr_: Some nights I sit down and think, "hey, I'm going to work on the docs tonight", and then I look at how much stuff needs to be done, and then I get discouraged. [00:40] ryah: ACTION is at 120% capacity unfortunately these days. [00:41] mjr_: ryah: other than presenting at various venues, what are you focusing on in node these days? [00:41] ryah: mjr_: working on internal stuff [00:41] ryah: that is, joyent stuff [00:42] mjr_: ohhh [00:42] mjr_: THAT kind of internal stuff [00:42] Tobsn: brb [00:42] ryah: yeah [00:42] creationix: alright, now my code is 60% node::Read [00:42] ryah: will get back to the other internal stuff soon [00:42] ryah: need to land that long stack branch [00:42] creationix: 5% node::Loop [00:42] tpryme: creationix: What's node::Read? - don't recall it from the docs [00:43] creationix: no clue, I'm guessing it's what fs.read calls internally [00:43] ryah: creationix: yes [00:43] ryah: er [00:43] creationix: ok, now down to 1m47s [00:43] ryah: it might be the _net.cc one [00:43] creationix: still slower than I would like [00:44] creationix: beats 4 minutes though [00:45] creationix: ryah: there is nothing I can do to make read faster is there? [00:45] creationix: it's all threadpool right? [00:45] Wandrewvious has joined the channel [00:46] creationix: interesting, also my process is using 178% cpu instead of 120% [00:46] creationix: must be the threadpool then [00:47] _announcer: Twitter: "#reddit Node.js: Javascript on the Server - Google Tech Talk by Ryan Dahl: submitted by jb55 [link] [comment] http://bit.ly/aqwHBb #rulez" -- REDDITSPAMMOR. http://twitter.com/REDDITSPAMMOR/status/19868721803 [00:47] creationix: 44% pthread_mutex_lock (inside node::Read) [00:47] creationix: and people say threads are free [00:48] creationix: too bad posix doesn't have non-blocking fs I/O [00:50] creationix: ryah: if my process if 44% pthread_mutex_lock maybe I'm calling fs.read too often? [00:50] creationix: it has 5 threads [00:51] creationix: (but I guess that's normal) [00:53] bleeber has joined the channel [00:53] bleeber: hello all [00:55] bleeber: we are currently using node.js to create a web based chat server for our online games. When run from the command line, it works fine for a while then crashes randomly. I then tried to add it to inittab so it would auto respawn but then it will not work. netstat shows the connections being established but you cannot chat. [00:56] bleeber: Has anyone else run into this issue and how would I go about working on a solution [00:57] creationix: bleeber: haven't seen that, but I did document my deployment http://howtonode.org/deploying-node-with-spark [00:57] creationix: not sure if that will help or now [00:57] creationix: *not [00:57] bleeber: creationix: anything could help at this point. Ill check it out [00:58] creationix: cool [01:01] bleeber: creationix: what is the benefit of using spark? [01:02] bleeber: creationix: nevermind, I just found the page [01:02] creationix: bleeber: it handles lots of cool deployment things without modifying the actual app [01:02] rauchg_: bleeber: it's weird that node `randomly` crashes, used to be common but not anymore, i'd check your actual app [01:03] visnup has joined the channel [01:08] creationix: bleh, reusing a buffer sucks when the function is called in parallel [01:08] creationix: I need a freelinst [01:08] creationix: *freelist [01:09] technoweenie has joined the channel [01:09] technoweenie has joined the channel [01:11] jashkenas has joined the channel [01:12] jashkenas has left the channel [01:12] tpryme: creationix: What's an example of a fn called in parallel in an async env like node? [01:14] creationix: tpryme: any async function that is called more than once before the first one comes back [01:14] creationix: it's quite common [01:14] tpryme: creationix: Oh, I see, like via recursion [01:15] creationix: tpryme: not usually recursion [01:15] tpryme: creationix: Oh, nm, gotcha [01:15] creationix: my case is I want to read two chunks from a file [01:15] franksalim has joined the channel [01:15] creationix: I fire off both queries at once, and they come back in arbitrary order [01:15] creationix: but since the function that does the fs read uses a global buffer, they clobber eachother [01:16] ryah: bleeber: you're listening for "uncaughtException" ? [01:16] bleeber: ryah: I finally figured it out and I am too embarrassed to say why [01:16] creationix: ryah: is freelist a public api? [01:16] ryah: damn. we really need --dev for development mode and default to producion mode [01:16] ryah: creationix: no [01:17] creationix: ok [01:17] creationix: ryah: default to production sounds good to me [01:18] ryah: production mode = never crash [01:18] rauchg_: bleeber: don't be embarrassed [01:18] ryah: just print stack traces [01:19] bleeber: When I was calling it from the command line I was in the directory. When I was calling it from the inittab I wasnt so it couldnt load all its deps [01:19] bleeber: lame [01:19] bleeber: sent all the output to a log and done [01:19] creationix: bleeber: yeah, that's why I cd to the folder in my upstart script [01:19] bleeber: yep [01:19] bleeber: oh well. works now [01:20] marshall_law has joined the channel [01:20] bleeber: and inittab keeps it running for now until we find out what makes it die [01:20] creationix: bleeber: I had a nasty error where my server wouldn't work from a browser, but works from curl [01:20] creationix: turned out that I was shelling out to gzip but only if the client accepted gzip encoding [01:20] keyvan has joined the channel [01:20] creationix: and gzip wasn't in my PATH [01:20] tmpvar: haha, nice [01:20] bleeber: fun fun fun. [01:21] tmpvar: thats pretty obscure [01:21] creationix: once I started logging errors it was pretty obvious [01:22] creationix: so to be safe, I put everything in my PATH explicitly in the upstart script and then cd to the directory [01:22] creationix: that seems to keep the env sane [01:25] tmpvar: i suppose you could also run a `which program` for all the things you'll need and store those off as well [01:25] tmpvar: (just thinking about deployment over many machines) [01:25] deepthawtz has joined the channel [01:26] tmpvar: and use __dirname ;) [01:29] reid1 has joined the channel [01:30] drudge: creationix: whenever i run spark 0.0.2 installed from npm, i get this: Error: Cannot find module '/Users/drudge/.node_libraries/.npm/spark/0.0.2/package/bin/spark [01:30] creationix: drudge: sounds like a npm issue [01:31] creationix: try uninstalling and re-installing from npm [01:31] _announcer: Twitter: "What a cool guy. Rolled the talk yesterday, and we can already see today. Node.js Google http://youtu.be/F6k8lTrAE2g" [pt] -- Éber F. Dias. http://twitter.com/eber_freitas/status/19871572595 [01:31] drudge: reinstalling npm or spark? [01:31] creationix: drudge: spark [01:31] creationix: it's independent from connect [01:31] drudge: same deal after a reinstall [01:32] bradleyprice has joined the channel [01:32] creationix: npm generates a shim that requires the real bin file [01:32] creationix: but for some reason it can't find it [01:32] creationix: spark by itself doesn't require anything except node internals [01:33] drudge: yeah running spark via /Users/drudge/.node_libraries/.npm/spark/0.0.2/package/bin/spark works [01:33] JimBastard has joined the channel [01:34] creationix: drudge: as a hack, find the spark shim (which spark) and replace it with a symlink directly to the spark file [01:34] coobr has joined the channel [01:34] creationix: it doesn't need a shim to work, symlinks work fine [01:34] WALoeIII has joined the channel [01:34] ditesh|cassini has joined the channel [01:34] drudge: creationix: thanks mate, hacky but it will do perfectly [01:35] creationix: drudge: oh, and file a ticket against npm [01:35] creationix: isaacs will probably fix it [01:35] creationix: right away [01:35] dmitry2 has joined the channel [01:37] drudge: filed creationix. http://github.com/isaacs/npm/issues/issue/135 [01:37] creationix: thanks [01:39] steadicat has joined the channel [01:41] creationix: drudge: ok, I talked to isaacs [01:41] creationix: you need to update your node for the shim to work [01:42] drudge: i'm on v0.1.101 [01:42] creationix: since my spark file doesn't end in .js older node can't require it [01:42] drudge: need 0.1.102? [01:42] creationix: not sure [01:42] creationix: I thought it was added earlier than 101 [01:42] creationix: could be wrong, it was a recent fix [01:43] creationix: I personally just use symlinks so I don't notice [01:43] _announcer: Twitter: "#nodejs + #expressjs has reignited my love for #js. All of the projects @ http://github.com/visionmedia/ rock!" -- Eric Clemmons. http://twitter.com/ericclemmons/status/19872379218 [01:43] drudge: yeah not a huge deal, definitely seems like a bug to me though [01:44] drudge: i'll upgrade to .102 and see if that changes anything [01:44] tyler_ has joined the channel [01:44] tyler_: can you use nodejs with google ajax api? [01:44] tyler_: javascript api [01:44] creationix: drudge: not sure it's a bug, but a bad combination of my file not ending in js, node not being able to require it and npm trying to require it [01:45] creationix: none of those alone are bad things [01:45] jamescarr: tyler_, yes [01:45] jamescarr: tyler_, you can use it with anything my friend [01:46] tyler_: jamescarr, how? they require you to
hello