[00:01] cloudhead has joined the channel [00:07] bentomas has left the channel [00:11] robrighter_ has joined the channel [00:18] technoweenie: http://github.com/technoweenie/ntest [00:19] technoweenie: probably going to js hell for the pseudo indentation style [00:19] binary42 has joined the channel [00:19] jcrosby has joined the channel [00:37] jed has joined the channel [00:51] jamiew has joined the channel [00:51] charlenopires has joined the channel [01:04] mikeal has joined the channel [01:15] Pilate has joined the channel [01:15] Pilate has joined the channel [01:16] kriszyp has joined the channel [01:19] LuigiMontanez has joined the channel [01:30] hassox has joined the channel [01:34] sixtus42 has joined the channel [01:35] mikeal has joined the channel [01:38] technoweenie: any suggestions on libs that provide enumerable methods on arrays and stuff? [01:39] jed: technoweenie: like this? http://github.com/documentcloud/underscore/ [01:39] technoweenie: this will work with node.js? [01:40] jed: technoweenie: i can't imagine it wouldn't. [01:40] DamZ_ has joined the channel [01:43] technoweenie: ah, i tried the tests and they didnt run [01:43] technoweenie: but the lib itself seems fine [01:43] technoweenie: i guess the tests would run fine in a browser [01:48] jed: technoweenie: yeah i think it assumes they're called from the html file. [01:48] technoweenie: yea thats why i didnt think this would work [01:49] jed: technoweenie: well, there's nothing really exotic in there, just a bunch of utils. [01:56] r11t has joined the channel [01:58] pjb3 has joined the channel [02:03] technoweenie: http://github.com/technoweenie/wheres-waldo still trying to get a hang of this whole async thing [02:03] technoweenie: i added some nodes talking about how im setting up various node/js libs as user-global libs [02:03] technoweenie: in ~/.node_libraries [02:04] RayMorgan has joined the channel [02:12] charlenopires_ has joined the channel [02:17] robrighter has joined the channel [02:36] erichocean: inimino: yt? [02:37] inimino: erichocean: yeah [02:37] erichocean: do you have any code showing how to traverse the parse trees created? [02:37] erichocean: I need to each for function calls with a specific name and replace them, and then output the modified code [02:38] erichocean: I could also use the streaming API if that's easier [02:38] erichocean: this is using your JS peg [02:38] inimino: ah [02:38] erichocean: i need to *s*earch for function... [02:39] inimino: there's a few things [02:39] inimino: let's see [02:39] inimino: http://boshi.inimino.org/3box/PEG/doc/parse_tree_representation [02:40] inimino: that's the first thing [02:40] inimino: that's probably enough to do basic stuff easily [02:40] erichocean: how would you suggest outputting the resulting code? [02:40] erichocean: just walk the tree? [02:40] jed has joined the channel [02:41] inimino: yeah [02:41] inimino: depending on how heavy what you're doing is, it might be worth trying to use some of the other stuff I have for dealing with trees [02:42] inimino: one thing is TAL, which is a DSL that looks like this: http://boshi.inimino.org/3box/PEG/PEG_codegen_5 [02:43] inimino: which is somewhat analogous to XSLT, in that it's a declarative language for programs operating on trees [02:43] inimino: it uses embedded JavaScript with a language that selects and binds nodes [02:44] rtomayko has joined the channel [02:44] inimino: but I haven't released or documented it yet, so ... [02:44] inimino: I don't recommend trying to use it yet [02:44] erichocean: one thing I'll be doing is outputting trees for Rascal: http://www.meta-environment.org/Meta-Environment/Rascal [02:45] erichocean: mainly so I can do dead code elimination [02:46] inimino: ok [02:47] inimino: yeah for that, probably a simple tree walker is easiest [02:47] inimino: read that text file that explains the format, and there's also an array attached to the parse function which contains the rule names [02:47] inimino: and it should be pretty straightforward [02:49] JimBastard has joined the channel [02:49] erichocean: great, thx, the parse_tree_representation docs have everything I need AFAICT [02:50] JimBastard: good morning party people [02:51] rtomayko_ has joined the channel [02:51] erichocean: inimino: do you have a grammar for the PEG format you accept? [02:52] inimino: erichocean: yeah: http://boshi.inimino.org/3box/PEG/PEG.peg [02:52] JimBastard: can anyone suggest the easiest way to ensure node will always be running on ubuntu? like if it shits itself it will get restarted? im kinda a linus noob [02:52] inimino: JimBastard: try monit [02:52] JimBastard: http://mmonit.com/monit/ ? [02:52] JimBastard: is it worth getting a product to do that? [02:53] JimBastard: instead of trying to do it natively? [02:53] inimino: yes [02:53] erichocean: daemontools works (and is free) [02:53] inimino: not sure what you mean by getting a product, it's just another daemon [02:53] JimBastard: aight i havent installed this in a while [02:53] JimBastard: let me try it out again, brb [02:54] inimino: ...monit is free [02:54] inimino: don't be confused by the nicely designed website into thinking it's not GPLed ;) [02:55] inimino: m/monit isn't [02:56] inimino: it's been a while since I looked at this site, or set up monit... [02:56] inimino: I set it up probably 5 years ago and never really had to mess with it after that [03:12] JimBastard: okay inimino i got it setup, now im gonna mess with it a bit to monitor node [03:17] bentomas has joined the channel [03:21] rtomayko has joined the channel [03:50] bry4n has joined the channel [04:06] pyrotechnick has joined the channel [04:06] pyrotechnick: how do i add a callback to posix.cat? [04:07] pyrotechnick: i can only get it working with .wait() [04:22] pyrotechnick has joined the channel [04:45] erichocean: this paper express much of why I'm not in favor of things like CommonJS: http://www.cl.cam.ac.uk/~srk31/research/papers/kell09mythical.pdf [04:46] erichocean: s/express/expresses/ [04:46] erichocean: and this is an example of an alternative I think is heading in the right direction: http://www.cs.utah.edu/flux/papers/knit-osdi00.pdf [04:52] erichocean: and I would say the argument for "better" linkers is similar to the argument for functional program (a new kind of "glue"): http://www.cs.chalmers.se/~rjmh/Papers/whyfp.pdf [04:52] erichocean: s/program/programming/ [05:10] ryah: erichocean: what was the point of the paper? [05:11] erichocean: that interfaces are information, and by importing them, you make software inflexible [05:11] erichocean: for example, I like your http_parser [05:12] erichocean: *because* it has no dependencies [05:12] erichocean: the problem with stuff like CommonJS is that you can't interpose imports [05:12] erichocean: say I import the redis library [05:12] erichocean: and it imports a logger library [05:13] erichocean: but I have my own logger library [05:13] erichocean: I have to modify redis to use mine [05:13] erichocean: and even if I do, the APIs might not match [05:13] erichocean: hence, even more integration work [05:14] erichocean: the situation is much worse in something like Python or Java where integration is part of the language itself [05:14] erichocean: epic fail [05:15] erichocean: and the leading cause of people rewriting the same damn code over and over and over [05:20] brosner has joined the channel [05:50] scudco has joined the channel [06:19] Yuffster_ has joined the channel [06:20] CIA-78: node: 03Ryan Dahl 07master * r5341359 10/ (src/node.cc test/mjsunit/test-eio-race4.js): [06:20] CIA-78: node: Fix another problem with the EIO interface [06:20] CIA-78: node: Should call eio_poll() when given a done_poll signal as well. [06:20] CIA-78: node: Bug report and test case by Kris Zyp - http://bit.ly/54pP4t [06:27] ryah: isaacs: ping [06:27] isaacs: erichocean: but Java programmers LIKE writing the same damn code over and over. [06:27] isaacs: ryah: pong [06:30] ryah: isaacs: your path changes - can you rebase them on top of master [06:30] CIA-78: node: 03Benjamin Thomas 07master * r73266cb 10/ (doc/api.txt src/node.js): Added extname function to path module that returns the extension. - http://bit.ly/53pmbx [06:30] ryah: isaacs: in particular 73266cb [06:30] isaacs: yep, just saw that [06:30] isaacs: kk, will do [06:30] isaacs: just a sec [06:32] pyrotechnick has joined the channel [06:35] ryah: ACTION wants to do a new release soon [06:36] isaacs: wait a sec, maybe i'm being dumb.. [06:36] isaacs: ACTION is a little afraid of rebase... [06:37] isaacs: ohh.... this is weird... i somehow managed to put your two commits on top of MY changes rather than the other way around. [06:41] ryah: :) [06:45] isaacs: ok, check out my master and see if that works for you [06:46] isaacs: i'm seeing this: http://skitch.com/isaacschlueter/nqi6r/node-master-sistertrain-lm-dev-js-node [06:47] isaacs: ah, when i rebased, it put my local master and remote master out of sync, and wouldn't let me push to github [06:51] isaacs: ok, force pushed onto my extname-and-basename branch [06:56] ab_ has joined the channel [07:07] ab_ has joined the channel [07:10] isaacs: ryah: http://github.com/isaacs/node/commits/extname-and-basename that work? [07:18] tiglionabbit: help me decide which of these web framework things I should use http://wiki.github.com/ry/node [07:24] joshthecoder: tiglionabbit, best bet is to try them yourself and see which one fits [07:25] tiglionabbit: I guess so. But I spend so much time evaluating libraries already, it's nice to be able to start with something decent [07:26] tiglionabbit: my first choice was picard, because it's a port of sinatra which I'm familiar with. But... upon trying it out, it doesn't seem to reload when I change things, or serve index files properly [07:28] martint-t has joined the channel [07:29] joshthecoder: they all seems to be a bit "green" still [07:30] isaacs: joshthecoder: that's kinda the way it goes. [07:30] isaacs: node itself is still in the 0.1.* area [07:31] tiglionabbit: the biggest disappointment for me is seeing that the v8 mongod drivers are hardly begun [07:31] tiglionabbit: mongodb I mea [07:31] tiglionabbit: n [07:38] tiglionabbit: what should I try next? [07:49] gwoo has joined the channel [07:56] scudco has joined the channel [08:01] joshthecoder: is there a REPL for node yet? [08:01] tiglionabbit: yes [08:02] tiglionabbit: node-repl [08:04] joshthecoder: ahh missed that [08:04] joshthecoder: thanks [08:07] tiglionabbit: also available as a library http://nodejs.org/api.html#_repl [08:34] qFox has joined the channel [08:36] ryah: github is so annoyingly slow. [08:38] tiglionabbit: wait a second here, the example and the table below don't match http://nodejs.org/api.html#_tcp [08:39] ryah: tiglionabbit: what do you mean? [08:40] tiglionabbit: socket.addListener("connect" ...) but the event is called "connection" [08:40] ryah: tiglionabbit: "connection" is an event that a server emits [08:40] tiglionabbit: oh [08:40] ryah: tiglionabbit: "connect" is for a socket [08:42] CIA-78: node: 03isaacs 07master * rb21056d 10/ src/node.js : [08:42] CIA-78: node: Rename path.filename() to path.basename() [08:42] CIA-78: node: Also make normalizeArray a bit more proper wrt handling dots when keepBlanks [08:42] CIA-78: node: is enabled. Fix indentation. - http://bit.ly/64JjXU [08:43] tiglionabbit: is there an example client to go with this example server? [08:46] jakswa has joined the channel [08:53] ryah: tiglionabbit: no, but if you make one i'll happily include it [09:02] tiglionabbit: hm, turns out it's very similar to the server [09:04] sudoer has joined the channel [09:15] tiglionabbit: ryah: is it really ok to start connecting before I've assigned my callbacks? [09:18] tiglionabbit: http://friendpaste.com/45YdMevzNhe6StedXF0hQe here's a silly client to test the server [09:18] tiglionabbit: though, I intentionally called tcp.createConnection without a port so I could apply the listeners before actually connecting [09:19] ryah: yeah it's okay [09:19] tiglionabbit: which causes it to print "(evcom) connect Can't assign requested address" sometimes [09:19] ryah: you can do new tcp.Connection() too [09:19] ryah: yeah you need to give it a port :) [09:20] tiglionabbit: no method Connection [09:22] ryah: oh yeah [09:22] ryah: oh well, just do createConnection [09:22] ryah: theres no race condition with adding listeners [09:24] tiglionabbit: why isn't there? [09:24] tiglionabbit: does it wait until the script is done executing to actually connect? Hm, couldn't in the case of the repl [09:24] tiglionabbit: or is this more of a "there's no way it could connect fast enough to break" [09:25] tiglionabbit: no, that can't be either, since anyone could break that by writing some slow code into the script [09:25] ryah: it takes at least one loop through the event loop to connect [09:26] bentomas has joined the channel [09:27] tiglionabbit: and it doesn't run the event loop until it gets to sleep? [09:27] teemow has joined the channel [09:29] CIA-78: node: 03David Sklar 07net2 * ra876df6 10/ (4 files in 4 dirs): Initial take on passing FDs between processes - http://bit.ly/5pNZSH [09:30] ryah: tiglionabbit: it doesn't try to connect until execution has passed voer the whole file [09:30] tiglionabbit: cool [09:31] tiglionabbit: wait, that can't be strictly correct since there are blocking calls like server.listen, right? [09:32] ryah: listen() is synchronous (for now) [09:32] ryah: connect() is not [09:32] tiglionabbit: nm, that call doesn't actually seem to block. [09:32] tiglionabbit: I put a puts after it and it printed right away [09:32] ryah: doesn't take so long to listen on a port :) [09:33] tiglionabbit: oh :P [09:33] tiglionabbit: when it said synchronous I thought it meant blocking forever [09:33] tiglionabbit: there are no functions in here that block forever, right? [09:38] ryah: no [09:38] ryah: there are very few functions which block for more than 1ms [09:39] tiglionabbit: how many concurrent users would you expect to support on one server? [09:40] ryah: depends on what they're doing [09:40] ryah: idling .. tens of thousands [09:45] tiglionabbit: how about receiving broadcasted data? [09:46] ryah: yeah - like ipv6 multi-messages? [09:46] ryah: i don't hvae support for that [09:46] ryah: mm i think [09:47] ryah: basically you're limited to simple tcp [09:47] ryah: nothing low-level - at least not in this version [09:49] tiglionabbit: hm. Does orbited broadcast stuff like that? [09:49] tiglionabbit: then again, my clients would likely need mostly different messages anyhow [10:06] sixtus42 has joined the channel [10:17] eviltwin has joined the channel [10:38] jakswa has left the channel [10:59] tiglionabbit: can you use the tcp connection stuff in a web browser? [12:11] eviltwin has joined the channel [12:26] sveisvei has joined the channel [12:55] spoob has joined the channel [12:59] un0mi has joined the channel [13:18] jasondavies has joined the channel [13:32] kriszyp has joined the channel [13:51] DamZ has joined the channel [14:06] DamZ has joined the channel [14:15] DamZ_ has joined the channel [14:48] jasondavies has joined the channel [14:59] felixge has joined the channel [15:13] micheil_mbp has joined the channel [15:20] jed_ has joined the channel [15:22] alex-desktop has joined the channel [15:38] binary42 has joined the channel [15:48] kriszyp: felixge: http://github.com/kriszyp/node-promise [15:49] felixge: kriszyp: ohhh nice [15:49] felixge: let me look at this now [15:50] felixge: the different ways to use it are confusing [15:50] felixge: what's wrong with a single way to use promises? [15:51] kriszyp: well, I was trying to make it backwards compatible for node [15:51] kriszyp: isn't that desired? [15:51] kriszyp: I thought that was one of the main needs for introducing this into node [15:52] felixge: well I'll have to study this closer. Are Deferreds and Promises the same here? [15:52] felixge: or what's the difference? [15:53] kriszyp: here, Deferreds are basically a type of Promise that is fulfillable [15:53] kriszyp: A promise provides .addCallback, .addErrback, .then, for listening for results [15:53] kriszyp: a deferred provides all that and callback, emitSuccess, resolve, etc. [15:53] kriszyp: callback = emitSuccess = resolve [15:54] kriszyp: .errback = .emitError = .reject [15:54] felixge: hm [15:54] kriszyp: callback/errback -> dojo, emitSuccess/emitError -> Node, resove/rejcet -> narwhal/ref_send [15:54] felixge: ok, but all of those behave the same? [15:54] kriszyp: yeah [15:54] felixge: ok [15:55] felixge: thanks for clarifying [15:55] felixge: so why do you separate Promises and Deferreds [15:55] felixge: I mean what is a Promise by itself good for? [15:55] ashb: The deferred object a promise object that provides a separation of consumer and producer to protect [15:55] felixge: > here, Deferreds are basically a type of Promise that is fulfillable [15:55] ashb: promises from being fulfilled by untrusted code. [15:56] kriszyp: yeah [15:56] felixge: why do we need both? [15:56] felixge: Would you ever want an unprotected promise? [15:56] spoob: please don't make promises into a nodejs version of monad, increasingly obscure and difficult to dscribe :) [15:56] kriszyp: heh [15:57] kriszyp: why would you ever want private access to a variable? it is good coding practice (and necessary for security) to only expose necessary functionality [15:57] felixge: I'm starting to get a brain freeze from all the promise stuff. WAY TOO MUCH names for the same concepts [15:57] felixge: confuses the hell out of me [15:57] felixge: I'd rather break BC than having to remember that 3-4 names describe the same concept [15:58] inimino: ACTION introduces an entirely new set of names [15:58] kriszyp: so the important thing is not how complicated are all the underlying CS principles, but how difficult is it to use [15:58] spoob: you could call it "hollywood" - "don't call me, I'll call you" [15:59] kriszyp: obviously, the entirety of the CS research on the topic is a difficult to completely comprehend [15:59] botanicus_ has joined the channel [15:59] inimino: hehe [15:59] kriszyp: the goal is certainly to make it simple to use [16:00] spoob: kriszyp; I just did the Hollywood version of the explanation.. "dude, it's like an agent, man... it's like, you don't call them, dude, THEY.... CALL.... YOU.....!" [16:00] jasondavies: is there a way to get the current module path from within a required module? [16:00] felixge: jasondavies: __filename [16:00] jasondavies: felixge: cool, thanks [16:00] felixge: or you mean the parents path? [16:01] felixge: kriszyp: exports.Promise = exports.Deferred = exports.defer = defer; [16:02] felixge: this makes me think there is no difference between a Promise/Deferred ? [16:02] kriszyp: right, there is just a single API [16:02] felixge: at least for people who use the API [16:02] felixge: so the difference you are talking about is just internally in the abstraction of the module [16:02] kriszyp: I could remove exports.Promise, I suppose [16:02] felixge: which kind of makes me question the value in abstracting that [16:02] jasondavies: felixge: that works fine, I just wanted the canonical path for a file in the same directory as the module so I just needed path.dirname(__filename) [16:02] kriszyp: I am open to suggestions [16:03] kriszyp: I am not sure what you want changed though [16:03] felixge: jasondavies: yip, exactly. I use that a lot [16:03] spoob: my sggestion is to call it Delegate, like a delegate object in Objective-C... (stirs the pot) [16:04] felixge: kriszyp: still going through the code, I'll be able to explain better in a sec [16:04] kriszyp: And I am not entirely sure that there is a strong CS historical basis for separation of terminology between Promise and Deferred [16:04] kriszyp: however, there is an obvious distinction between producing a promise and consuming a promise [16:04] felixge: spoob: Delegate is kinda cool, its one of the things I like in obj-c [16:04] spoob: felix; yes, it also seems to be a conceptually simple way of handling deferred or streamed I/O [16:05] felixge: spoob: but a delegate is an object you reference, a promise is the async-counterpart to a regular function call [16:05] felixge: so I don't think the concept maps 1:1 [16:05] spoob: doesn't a promise have a callback when it has more I/O? [16:05] felixge: spoob: no, a promise only computes a single value [16:06] felixge: then its "done" [16:06] felixge: and its only purpose is to keep the computed value around for anybody interested in it [16:06] technoweenie has joined the channel [16:07] spoob: like next in a Scheme stream? [16:07] felixge: spoob: not familar with that, sorry [16:07] spoob: it returns a function that computes a value, and optionally another function that will compute the next value... to create stream I/O in Scheme [16:08] felixge: spoob: I think there is a similarity here, yeah [16:09] felixge: promise callbacks create new promises [16:09] felixge: which is similar [16:10] kriszyp: anyway, I have tested the code, but I haven't tried actually replace process.Promise in Node yet [16:10] kriszyp: and I think the error handling should match what you had asked for [16:10] kriszyp: at least close, I think [16:10] kriszyp: :) [16:12] sveisvei has joined the channel [16:13] felixge: kriszyp: why do you define the Deferred methods inside the constructor? [16:14] felixge: so you can have private variables? [16:15] kriszyp: yes [16:15] felixge: k [16:15] lattice has left the channel [16:15] kriszyp: meets the ocap needs [16:15] felixge: so the Promise class in your code is just an abstract part of the API, but the Deferred is the implementation [16:16] lattice has joined the channel [16:16] kriszyp: yes [16:16] felixge: yeah, that was confusing to me, but I guess it doesn't matter [16:17] kriszyp: yeah, a user shouldn't need to consider any difference between a deferred and a promise, it is more an implementation organization [16:17] kriszyp: they just new process.Promise() to create a promise [16:17] spoob: don't module systems in commonjs define their own namespace so there's no need to make variables private? [16:17] felixge: why do you create a new promise in the Deferred constructor if the Deferred itself is a Promise? [16:18] kriszyp: and naturally it will have emitSuccess/callback, but the promises that are returned from addCallback will not (because they are fulfilled by the chain, not directly) [16:18] kriszyp: so that you can access the consumer-only promise from the "promise" property [16:18] kriszyp: I could remove that [16:19] kriszyp: But that is what was demanded in narwhal for the ocap requirements [16:19] felixge: hm [16:19] felixge: ok [16:20] felixge: I'm ok with promises being not incredibly fast, I think callbacks should be used where that is the main concern, so that's ok [16:20] felixge: :) [16:21] kriszyp: spoob: I am not sure I understand how namespacing provides privatization [16:21] kriszyp: you mean their own scope? [16:22] felixge: kriszyp: I think I like your implementation : ) [16:22] kriszyp: well you should try it out before you commit to liking it ;) [16:22] felixge: I will : ) [16:23] kriszyp: the error handling is tricky, and I am hoping I understand your intent correctly [16:24] kriszyp: p = new process.Promise(); p.addCallback(function(){ throw new Error("won't be caught"); }); p.emitSuccess("done"); // should throw an uncaught error [16:25] felixge: kriszyp: I think you do understand [16:25] felixge: at least your code should do what I want [16:25] felixge: :) [16:25] felixge: kriszyp: yes [16:25] felixge: the above example is correct [16:25] kriszyp: p = new process.Promise(); p.addCallback(function(){ throw new Error("will be caught"); }).addErrback(function(){ /* caught */}); p.emitSuccess("done"); // should be caught [16:25] felixge: yes [16:25] felixge: that looks perfect [16:25] felixge: I think this is how it should be [16:26] felixge: the only problem is that if you perform aysnc I/O in the callback and throw an error after that finishes, it won't be caught [16:26] felixge: so you'd have to manually catch that [16:27] felixge: and emit an error [16:27] spoob: bloody heatwave. my laptop power supply is cooking itself. It's 40 degrees Celsius (104 F). I hear it's cold over there for you chaps? [16:27] felixge: kriszyp: one thing, you enqueue all callbacks? [16:27] kriszyp: manually catch it? [16:27] kriszyp: felixge: yes, do you object to that? [16:27] felixge: spoob: cold? Yeah -10 C here [16:28] spoob: I guess you've seen snow then. I've never seen snow in my life ;) [16:28] felixge: kriszyp: it seems inefficent for no good reason. The only part that needs enqueue is reject [16:28] felixge: spoob: where are you located? [16:28] spoob: south australia [16:29] felixge: spoob: not going to feel sorry for you, I'd trade ;) [16:29] kriszyp: I've haven't seen my whole lawn in a month or so (under the snow) [16:29] spoob: so when my laptop power supply catches on fire, you'll go "no sympathy"? :) [16:30] kriszyp: felixge: I thought Node users appreciated enqueuing, to spread tasks out [16:30] kriszyp: and I thought the event loop was pretty efficient [16:30] technoweenie: hey is there a way to have promises with subpromises? [16:31] kriszyp: yes [16:31] technoweenie: i have a method that fires off a couple redis calls, and i want the returned promise to emit success when all the redis calls are done [16:31] felixge: kriszyp: well, I can't speak for everybody - but I like simple & efficient behavior. Putting something in the event loop that could be done right away seems wrong [16:31] kriszyp: yes, this module handles "subpromises" fine [16:31] technoweenie: http://gist.github.com/272972 [16:31] kriszyp: well, it easy to turn off enqueuing [16:31] felixge: technoweenie: I think you mean grouping promises [16:32] kriszyp: I haven't heard a lot of arguments one way or the other [16:32] technoweenie: yes groping promises [16:32] technoweenie: grouping [16:32] spoob: a very slight increase in complication will result in an exponential number of questions on the mailing lists. I think Haskell's choice of "monads" will kill them in the end. [16:32] felixge: kriszyp: I'm not 100% sure either, I'm just guessing here [16:32] technoweenie: ah ha finding some google hits on that [16:32] felixge: technoweenie: do you need to access your redis from outside your node app? If not I got something better for ya :) [16:33] technoweenie: did you write dirty or something [16:33] felixge: yeah [16:33] felixge: :) [16:33] technoweenie: yea i thought your nick looked familiar [16:33] felixge: Will release it on tuesday [16:33] technoweenie: well i could probably make it work with dirty too, that's a good idea [16:34] kriszyp: technoweenie: one could certainly write a helper function to make it easy to do that [16:34] felixge: technoweenie: I'll add documentation later tonight, will ping you if you're around [16:34] technoweenie: ok cool [16:34] felixge: kriszyp: yeah, I think your new promise stuff handles sequential promises really well [16:34] technoweenie: yea i'm trying to write a join method. i failed (the paste was my only working sample). http://www.dashdashverbose.com/2009/11/joining-promises-for-parallel-rpcs-in.html has something though [16:34] felixge: but parallel promises should have a solution as well [16:34] felixge: in the core [16:35] spoob: top hit for "nodejs dirty" is Use this clone URL yourself. git clone git@github.com:fwg/nodejs.irc.git .... yes, make sexy time. dirty dirty. FIXME facetious variable name! ... [16:35] kriszyp: in the core? [16:35] kriszyp: I could add a utility function for that if desired [16:36] kriszyp: although a minute ago it sounded like you wanted the module to be trimmed ;) [16:36] kriszyp: it is certainly trivial to compose on top of the currently functionality though [16:36] felixge: spoob: https://github.com/felixge/node-dirty [16:37] felixge: spoob: http://github.com/felixge/node-dirty/blob/master/example/basic.js is probably the best intro [16:37] spoob: I get the key value part, but why is it dirty? [16:38] felixge: kriszyp: Pretty every 2nd project I've done in node so far had to deal with promise grouping, so I think it should be included with the batteries [16:38] kriszyp: k [16:39] kriszyp: so typically with lists of promises, there is a function to returns promise for when all are finished, and when the first one is finished [16:39] kriszyp: any suggestions for names? [16:39] felixge: spoob: NoSQL is for adults. It solves performance problems by shifting more work to the clients. Dirty is for kids, it makes working with NoSQL a lot easier by embedding the db into your process and allowing you to iterate over documents in memory directly [16:40] kriszyp: every and some to mirror Array.prototype? [16:40] technoweenie: oh i thought dirty was for addults [16:40] technoweenie: adults [16:40] spoob: kriszyp; reduce? [16:40] felixge: technoweenie: well, yeah there is an overlap :) [16:40] felixge: kriszyp: I think 'group' [16:40] spoob: kriszyp; present? [16:41] felixge: require('promise').group(new process.Promise(), new process.Promise()).addCallback() [16:41] technoweenie: hey isn't there a magic variable in a function that's the array of arguments [16:41] inimino: spoob: I really recommend learning Haskell :D [16:41] technoweenie: 'arguments' seems to be a hash? [16:41] spoob: inimino; :) [16:41] kriszyp: and it the fulfilled value is an array of the fulfilled values from the each promise, right? [16:41] lattice: felixge: if it doesn't already, you should build a way for dirty to fall back to a NoSQL store if the key isn't present locally. That would be awesome. [16:41] felixge: technoweenie: arguments is a hash that disguises as an array [16:42] felixge: technoweenie: try: var args = Array.prototype.slice.call(arguments); [16:42] inimino: technoweenie: it's not a hash, or an array [16:42] inimino: technoweenie: https://developer.mozilla.org/En/Core_JavaScript_1.5_Reference/Functions_and_function_scope/Arguments [16:42] felixge: lattice: that will be totally doable. The idea is that dirty is the "minimal viable key value store", and you can quickly build "your perfect key value store" on top if it within minutes. [16:43] felixge: lattice: I will work on some plugins that add stuff like a JSON API or your suggestion [16:43] felixge: but they'll be strictly optional [16:43] spoob: I'm all in favour of minimal, low conceptual overhead, foundations for the more capable stuff on top [16:43] spoob: i'm reading this Promise stuff and thinking "it's OK if I can use callback instead" :) [16:43] felixge: spoob: dirty has no conceptual overhead. Documents are serialized into JSON and added to a file on disk seperated by newlines [16:44] lattice: felixge: I was thinking something like dirty.addCallback('missingFetch', function (arg) { noSQL.fetch(arg) } [16:44] felixge: writing to disk happens every x-Records or every x-Seconds [16:44] felixge: lattice: Accessing and storing objects in dirty is sync. [16:45] felixge: db.set('foo', 'bar') will make 'foo' available right away [16:45] felixge: it might take a moment to sync to disk [16:45] felixge: but you won't have to wait for a callback if you want to do db.filter(...) right after [16:45] felixge: if you want async fetching/setting than that will be a project on top [16:45] felixge: Like dirty-couchdb [16:46] felixge: or whatever [16:46] spoob: yes, it looks like something I want to use :) [16:46] felixge: dirty-rest [16:46] felixge: dirty-tcp [16:46] felixge: etc. [16:47] lattice: huh, okay. So what's the difference between dirty and a global anon. object? [16:47] felixge: I mainly like that you can take it any direction you want / need [16:47] felixge: you start with the simplest possible key value store that can get your project of the ground [16:47] felixge: and makes development easy [16:47] felixge: and then you make it more capable as you need to [16:48] felixge: lattice: that dirty persists to disk for you automatically [16:48] felixge: lattice: it also makes iteration of items fast (like 33 mio iterations / sec fst) [16:48] felixge: you only get this performance with Arrays in JS [16:48] felixge: (v8) [16:48] felixge: Array.filter [16:48] felixge: but you cannot access arrays by random keys [16:48] felixge: so dirty keeps a hash map that maps your keys to the array indexes [16:48] felixge: so you get the best performance out of both worlds [16:49] lattice: *nod* - but storage is pretty much the first thing I'd want to be async, and the thing I really really like about node.js is that it strongly emphasizes the callbacks-everywhere approach [16:49] felixge: Unfortunately Array.push() is very slow in v8, so you "only" get ~100k writes / sec [16:49] inimino: felixge: is filter optimized, over using a loop? [16:49] inimino: in V8? [16:49] spoob: when it's that fast, sync vs async is not an issue [16:49] felixge: lattice: yes, the storage is async, if you want to wait for you doc to be stored on disk, you do: db.set('foo', {}, function() ...) [16:49] technoweenie: promise group implementation: http://gist.github.com/272972 [16:50] felixge: inimino: I think filter is optimized like hell, I wasn't able to build something as fast myself [16:50] inimino: wow, awesome [16:50] lattice: technoweenie: nice [16:50] kriszyp: technoweenie: you don't want the result of the promises? [16:50] kriszyp: that doesn't seem useful [16:50] felixge: but to be realistic, when actually filtering (that means accessing properties of your document and returning true/false 50% of the time) you only get ~10 mio. reads /sec [16:50] felixge: but that's still fast, right? :) [16:51] technoweenie: kriszyp: oh not in my specific case, no [16:51] inimino: hehe, yes [16:51] kriszyp: http://gist.github.com/272983 [16:51] lattice: felixge: sure, but async reads are actually incredibly important. for any network-based read on node.js, they *have* to be async, or you're going to block all pending requests [16:51] kriszyp: not tested at all... [16:51] felixge: lattice: dirty does not do any networking [16:52] felixge: lattice: http://github.com/felixge/node-dirty/blob/master/lib/dirty.js [16:52] felixge: it's only 160 lines of code [16:52] kriszyp: also handles values that are not promises (mixed arrays of values and promises) [16:52] felixge: I keep it to the bare minimum [16:52] felixge: that is persisting a key-value map to disk periodically [16:52] felixge: and the ability to reload it from disk [16:52] kriszyp: and it handles errors [16:53] felixge: kriszyp: sorry for bailing out of the promise discussion for a moment here, I'm still digesting my thoughts on it [16:53] kriszyp: np [16:53] kriszyp: no hurry [16:53] kriszyp: probably should get some other stuff done [16:54] lattice: felixge: right, I know – I'm saying that for the api, if it's designed to do a callback, then you can transparently switch the back-end, or use dirty as a universal NoSQL cache / interface without having to re-write all your code. [16:54] felixge: lattice: I guess I could add a callback parameter to the get method [16:55] felixge: lattice: but filter ... you won't get the same flexibility with it when you filter on a remote node [16:55] felixge: you can't access the variables from the global scope [16:55] felixge: I kind of want people to rewrite their code at some point [16:55] felixge: they should start with the easiest thing [16:56] spoob: if you made a callback for pre-fetching the data, so that accessing the data was guaranteed not to block, then you'd have the best of both worlds [16:56] felixge: spoob: a callback for pre-fetching? [16:56] lattice: spoob: what felixge said ;-) [16:57] spoob: at the moment, you can't access data without potentially blocking on network access to a backend [16:57] spoob: if the code is written to use that API, then switching in the back end will need rewriting the code [16:57] spoob: however this is because of accessing data that is possibly not yet loaded from the backend [16:58] felixge: spoob: you are always blocking the event loop when you evaluation a javascript expression [16:58] felixge: but I get what you mean [16:58] spoob: if, instead, you did a "load data" with a callback, then you would be able to access the data without doing callbacks after that point [16:58] johan-s has joined the channel [16:58] felixge: spoob: but what's the point of loading the data from somewhere else? [16:58] lattice: felixge: because you have more than one server? [16:58] spoob: so that you can switch in a network backend without rewriting code [16:59] felixge: lattice: why would you have more than one server? [16:59] felixge: for redundancy? [16:59] lattice: felixge: I hope that's a joke. :-) [16:59] inimino: spoob: how is that different from what it does now...? [16:59] lattice: spoob: why do a pre-load callback, rather than just having get() take a callback by default? [16:59] felixge: lattice: no. The biggest reason for adding a dirty server would be running out of memory. In that case it wouldn't be practically anymore to load another servers database into memory [16:59] spoob: lattice; because felixge says that he wants to access the data from within a filter [17:00] technoweenie: ok i have PromiseGroup returning values [17:00] inimino: spoob: currently it 1) loads the data from disk, and then 2) allows non-blocking access to it, since it's in memory [17:01] spoob: inimino; when the backend gets switched to something else, then presumably not all of the data would be loaded. lattice points out that will need a rewrite of code since the API assumes there is no fetch delay [17:01] spoob: so I'm suggesting that instead of every get() having a callback, and also not pre-loading everything, felixge could add in an API for loading specific items of data [17:01] felixge: hm [17:02] felixge: well, I'm not sure if that would need an explicit API [17:02] felixge: seems like it could be handled in userland code pretty easily [17:02] lattice: spoob: ahh, i see, just for the filter. glanced away, just catching up [17:02] inimino: spoob: ah, I thought by "load data" you meant loading all the data [17:02] inimino: spoob: yeah, that would work but the API is simple by design, I think it fills a good niche this way [17:02] felixge: I have sort of a vision for dirty [17:02] lattice: why couldn't filter do the same as get and delegate to a "filter" method on a networked NoSQL (or SQL, if you want) backend? [17:02] lattice: e.g., couch's search method [17:03] inimino: spoob: something that needs async reads probably isn't going to fit with what felixge is doing [17:03] felixge: "Dirty" means that you start your app by explicitly acknowledging its inability to scale beyond a single server. In exchange you get a huge conceptual simplification to everything you are doing and the project launches in a few weeks. [17:03] inimino: but I think there's room for both, with different APIs and different use cases [17:03] lattice: it's not going to be as fast as Array.filter, but honestly, that doesn't matter when your database is 10 terabytes and it's either "Not happening" or "Slower than Array.filter" [17:04] felixge: once you need to spread out, dirty makes that easy by having a simple file format (so you can migrate to a more serious NoSQL solution) [17:04] felixge: or you can build upon it, and add network APIs [17:04] lattice: felixge: I totally disagree. The core theory behind node is that everything is async, and that the language support is clean from the get-go. [17:04] dks_ has joined the channel [17:04] felixge: APIs that actually represent your application, not a vanilla key-value store [17:05] felixge: so you'd have several nodes that have a rest interface like: '/users/find' [17:05] cloudhead has joined the channel [17:05] sveisvei: felixge: implied global in dirty.js on line 7 [17:05] lattice: Would it be possible to get the same effect by doing some kind of asynchronous assignment operation? [17:05] felixge: in your application you should abstract your business logic into models anyway and make those return promises [17:05] lattice: var x := dirty.get('key'); [17:05] felixge: you just write a very light mapper on top of dirty [17:06] spoob: i predict that promises will be the monads of nodejs... [17:06] technoweenie: extracted PromiseGroup out as a module: http://github.com/technoweenie/wheres-waldo/blob/master/lib/promise-group.js [17:06] felixge: sveisvei: thanks, will fix [17:06] lattice: where := means "implicitly wrap that in a promise, and return the value back when it's done" ? [17:06] inimino: lattice: like a promise? ;) [17:06] lattice: inimino: but without specifying a callback function, just have it inline. [17:07] lattice: I don't know the v8 internals, so I don't know if this is possible at alll [17:07] felixge: sveisvei: fixed [17:07] inimino: I don't see how that fits with the JavaScript execution model at all, V8 or no [17:07] felixge: I think if you want to go 'async' with dirty I/O right away, you should do a layer on top of dirty. [17:08] felixge: Dirty is dirty for a reason :) [17:08] inimino: spoob: Promises are not nearly as useful as monads [17:08] felixge: and that layer would ideally be your business logic [17:08] inimino: spoob: or as general [17:09] spoob: inimino; I mean in question generating flooding mailing lists type equivalence. ;) [17:09] lattice: felixge: eh, I think dirty's get api could be async and still useful. ;-) [17:09] felixge: lattice: I'll support it. But maybe as a plugin [17:10] spoob: inimino; I'm already concoting "promise what is on first base, why promise is on second base" jokes [17:11] spoob: "who?" "no, that's promise who not who promise" [17:11] inimino: spoob: perhaps, but I doubt it [17:16] lattice: technoweenie: you could take a hash and have named promises, instead of just positionals [17:17] technoweenie: hmm i like that [17:17] grantmichaels has joined the channel [17:25] lattice: technoweenie: something like this (untested): http://gist.github.com/272992 [17:26] lattice: should work equally well with arrays / hashes [17:27] technoweenie: what happens if one of the promises finishes before you're done setting up [17:27] lattice: technoweenie: hehe - i just fixed that, reload the gist [17:28] grantmichaels has joined the channel [17:41] dnolen has joined the channel [17:44] grantmichaels has joined the channel [17:45] eddanger has joined the channel [17:51] jed_ has joined the channel [18:04] grantmic_ has joined the channel [18:04] martint-t has joined the channel [18:20] dnolen has joined the channel [18:27] technoweenie has joined the channel [18:47] evilclone has joined the channel [18:50] jamiew has joined the channel [19:18] bry has joined the channel [19:18] felixge: ryah: you around? Any thoughts on the sync require patch? The new libeio fix still doesn't seem to address my reloading issue :| [19:21] okito has joined the channel [19:32] okito has joined the channel [19:43] brosner has joined the channel [19:57] jamiew_ has joined the channel [19:58] jamiew has left the channel [20:03] okito has joined the channel [20:05] pjb3_ has joined the channel [20:06] pjb3_ has joined the channel [20:08] jed__ has joined the channel [20:11] felixge has joined the channel [20:16] micheil has joined the channel [20:18] felixge has joined the channel [20:19] ryah: felixge: ping [20:19] felixge: ryah: pong [20:19] ryah: felixge: i haven't had a chance to look at the patch yet [20:19] felixge: np [20:19] felixge: any look with your house hunt? [20:20] ryah: meh. not yet [20:20] felixge: when does your job start? [20:20] ryah: next week, but the first week is just a big meeting [20:20] ryah: so really the week after [20:21] felixge: you haven't heard anything about it yet, is it for the CIA ? :) [20:22] ryah: the cia of ssjs :) [20:22] felixge: hehe [20:22] felixge: ryah: what company is it, and what's your job ? [20:22] sixtus42 has joined the channel [20:23] ryah: joyent [20:23] ryah: going to have lots of time to work on node [20:23] ryah: they're really big on js [20:24] felixge: ah I think you mentioned it before (google showed the link as visited *g*) [20:24] felixge: ryah: anyway, can you send me the stuff you'd like to get mentioned in the talk at some point soon, trying to finish the presentation [20:25] okito has joined the channel [20:25] ryah: ah right [20:25] sixtus42: felixge: what talk? [20:25] sixtus42: ryah: congrats [20:26] sixtus42: ryah: are you aware of anybody doing node in strongarm? [20:26] sixtus42: I mean on strongarm [20:29] jed__: ryah: yeah, congrats! [20:32] felixge: sixtus42: berlin JS usergroup meetup on tuesday, you in? [20:32] sixtus42: felixge: link? [20:32] tiglionabbit: so, how do you stream data to a browser with ajax? [20:33] tiglionabbit: do you use the http or tcp module in nodejs? What do you use in the browser? [20:33] felixge: sixtus42: http://groups.google.com/group/js-berlin/browse_thread/thread/a6e4d775935241f0 [20:35] sixtus42: felixge: good thing you reminded me, that wasn't on my radar anymore. yeah, I do think I'll snow by ;-) [20:36] ryah: sixtus42: nope [20:36] sixtus42: ryah: well, I guess I will try then, once I figure out how to get gcc up and running on an n900 [20:37] sixtus42: ACTION wondering if 96 actually was the last time I coded C.... that'll be fun [20:38] felixge: sixtus42: cool! see you there [20:41] tiglionabbit: the chatroom example uses jquery which reconnects after each message. Is this the recommended way to do it? Is there a way to do it without reconnecting constantly? [20:42] tiglionabbit: divert me to documentation if I'm missing anything [20:45] sixtus42: tiglionabbit: http://en.wikipedia.org/wiki/Comet_(programming) [20:46] tiglionabbit: sixtus42: why are you linking me to a general article? Is there something wrong with my question? [20:47] sixtus42: tiglionabbit: well, the implementation is driven by the browser, not by node.js (which is way more powerful) [20:47] jamiew_ has joined the channel [20:47] sixtus42: tiglionabbit: the article describes your options (node.js can do all of them) [20:47] jamiew_ has left the channel [20:47] tiglionabbit: sixtus42: yeah but there are client side javascript libraries like the one bundled with orbited that receive data without hanging up, right? [20:48] CIA-78: node: 03Ryan Dahl 07master * rc9e2143 10/ (7 files in 2 dirs): Upgrade http-parser - http://bit.ly/8WP75Y [20:48] CIA-78: node: 03Ryan Dahl 07master * r39ca935 10/ (ChangeLog doc/api.txt doc/index.html wscript): bump version - http://bit.ly/7kMLny [20:51] tiglionabbit: I'm worried that data could be lost between connections, if it were served up quickly [20:54] tiglionabbit: ryah: what do you think? [20:55] erichocean: hmm, none of the node.js WebSocket libraries correctly handle message framing from the client :-( [20:56] tiglionabbit: framing? [20:58] erichocean: http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-46#section-1.3 [21:02] ryah: tiglionabbit: sorry got to go [21:02] tiglionabbit: =[ [21:04] RayMorgan has joined the channel [21:07] erichocean: http://blogs.webtide.com/gregw/entry/how_to_improve_websocket is a good overview of outstanding WebSocket issues [21:10] cloudhead has joined the channel [21:11] cloudhead has joined the channel [21:12] RayMorgan has joined the channel [21:14] cloudhead has joined the channel [21:25] robrighter has joined the channel [21:38] r11t has joined the channel [21:39] DamZ has joined the channel [21:40] tiglionabbit: hum, maybe my question doesn't matter that much after all. The chat client makes use of incoming timestamps to decide what data it needs to send clients, which is probably going to be more reliable than just sending off messages whenever they're available... but if everyone's listening on a different event stream, I'd have to keep a backlog for every user individually. crap. [21:48] rictic has joined the channel [21:59] mikeal has joined the channel [22:02] jed has joined the channel [22:07] tlrobinson has joined the channel [22:11] grantmichaels has joined the channel [22:19] sktrdie has joined the channel [22:19] sktrdie: hi [22:20] sktrdie: node js uses the Event model to handle I/o, but i dont understand how each request can coexist in the same thread? [22:20] sktrdie: what if a request takes a long time to run [22:20] sktrdie: i know it's non-blocking [22:20] sktrdie: but it can still do some math thing, that takes a long time [22:20] sktrdie: (not I/O) [22:21] sktrdie: i guess i'm talking about the http server [22:21] ryah: sktrdie: yeah, if you did while(true) {;} it would block [22:21] sktrdie: all the other requests as well?? [22:22] sktrdie: i guess this model is only to be used for special cases [22:22] ryah: yes [22:23] sktrdie: ok well, what about non blocking functions? [22:23] sktrdie: where are the processed? [22:23] sktrdie: in another thread? [22:23] sktrdie: s/the/they [22:24] ryah: if you have a lot of math to compute, it's probably best to move it to a different process [22:24] ryah: then do ipc between the http server and the computing process [22:24] sktrdie: no i mean, like the the I/O stuff [22:24] sktrdie: in nodejs is computed somewhere else, right? [22:24] sktrdie: instead of the HTTP process [22:25] ryah: no - it's done all in the same thread [22:25] sktrdie: are tey all computed in the same thread? [22:25] sktrdie: the same thread of the HTTP request?? [22:25] ryah: yes [22:25] sktrdie: what? [22:25] sktrdie: hrm [22:25] sktrdie: i thought it would be logical to have another one [22:26] sktrdie: why would you mix the both? [22:26] ryah: well, the way posix non-blocking i/o works is that you just send(data) [22:26] sktrdie: libevent i think uses another thread [22:26] ryah: in the same thread [22:27] ryah: and maybe it works and maybe it doesn't (because the kernel send buffer is full) [22:27] ryah: if it doesn't then you ask the kernel to give you a callback when the fd is writable again [22:27] ryah: then you try it again [22:27] sktrdie: so if it's in the same thread as the Request, when does it start running the I/O? [22:28] sktrdie: like the actual read() for example [22:28] sktrdie: sorry, im not that proficient with concurrency [22:28] ryah: well, depends on which read() you mean... [22:29] sktrdie: the posix one? [22:30] ryah: i mean which call to read() you mean. nodejs is calling read() all the time [22:31] sktrdie: well, say you listen for a database query [22:31] sktrdie: you're saying that the query will run in the same thread [22:32] ryah: ideally, yes [22:32] sktrdie: but, if also other requests are served by the same thread [22:33] ryah: node multiplexes the i/o using select() [22:33] ryah: you should read "man select" [22:34] ryah: (it actually doesnt use select, but which ever i/o multiplexer is best for each platform) [22:34] ryah: (but same idea) [22:35] sktrdie: hrm [22:36] sktrdie: so where does it block and listen for read status? [22:37] ryah: on the call to select() [22:37] sktrdie: and where is it? [22:37] ryah: where in the node source? [22:38] sktrdie: i mean conceptually [22:38] sktrdie: it can't be in the same HTTP request thread [22:38] ryah: yes, it is [22:39] sktrdie: ok [22:39] ryah: the call is here: http://github.com/ry/node/blob/master/src/node.js#L988 [22:40] ryah: node uses multiple threads internally for things that cannot be made non-blocking. like file i/o [22:43] sktrdie: does it use the system stuff from v8? or is it built in nodejs? [22:43] sktrdie: like system calls [22:44] sktrdie: i guess v8 just gives you the bare essentials, and you have to build the rest. [22:50] tav_ has joined the channel [23:16] felixge has joined the channel [23:25] webben has joined the channel [23:52] mikeal has joined the channel