What, you may ask, is js-ctypes? Let’s say you’re writing a Firefox extension in JavaScript that needs to call into native code. (For example, weave-crypto, which needs to call into the NSS library. Or your extension here, which, say, wants to call into NSPR, libc, or Win32 functions directly.) Currently, you’re limited to either a) using the scriptable XPCOM interfaces provided by libxul, or b) writing and implementing your own XPCOM interfaces and shipping binary code with your extension. If a) is inadequate, you’re stuck with b), which makes shipping an extension much harder — binary code must be built for each supported platform, and rolled up together into your cross-platform xpi.
Thusly, the answer is js-ctypes: it allows JavaScript to call into native C code and manipulate data types, without using XPCOM and without compiling a line of code. This means you don’t need to define XPCOM interfaces, and can use shared libraries like libc directly. As a side benefit, much of the type conversion overhead of XPConnect is avoided, so that execution can be faster. (I’ll be benchmarking in a later post.) This will ship with Gecko 1.9.3, which will be underpinning Firefox 4.
But how, you say? To the examples (tested on 32-bit x86 Linux; sans cross-platform safety):
1. Opening a library.
// First, import the ctypes module.
Components.utils.import("resource://gre/modules/ctypes.jsm");
// Open libc.
let library = ctypes.open("libc.so.6");
// Declare the 'fopen' function, which has a C prototype of:
// FILE* fopen(const char* name, const char* mode);
let fopen = library.declare("fopen", // symbol name
ctypes.default_abi, // cdecl calling convention
ctypes.StructType("FILE").ptr, // return type (FILE*)
ctypes.char.ptr, // first arg (const char*)
ctypes.char.ptr); // second arg (const char*)
// Call the function, and get a FILE* pointer object back.
let file = fopen("hello world.txt", "w");
// What is 'file'?
file.toString();
===> "ctypes.StructType("FILE").ptr(ctypes.UInt64("0x9781b38"))" // pointer value
// ... write data to file here ...2. Declaring and using structs and arrays.
// Declare the 'hostent' struct, which contains five fields, each with their own
// types. This corresponds to the C declaration:
// struct hostent {
// char* h_name; // name of host
// char** h_aliases; // array of strings representing aliases of the host
// int h_addrtype; // whether the IP address is IPv4 or IPv6
// int h_length; // length, in bytes, of the IP address
// char** h_addr_list; // array of IP addresses from name server
// };
let hostent = ctypes.StructType("hostent",
[{ h_name : ctypes.char.ptr },
{ h_aliases : ctypes.char.ptr.ptr },
{ h_addrtype : ctypes.int },
{ h_length : ctypes.int },
{ h_addr_list : ctypes.uint8_t.array(4).ptr.ptr }]);
// Declare the 'gethostbyname' function, which has a C prototype of:
// struct hostent* gethostbyname(const char* name);
let gethostbyname = library.declare("gethostbyname",
ctypes.default_abi,
hostent.ptr, // use our 'hostent' type
ctypes.char.ptr);
// Ask our function to resolve a hostname.
let google = gethostbyname("mail.google.com");
// Dereference the pointer to 'hostent' struct, access the 'h_name' field, and
// convert it to a JS string. This is roughly equivalent to the C expression:
// printf("%s", google->h_name);
google.contents.h_name.readString();
===> "googlemail.l.google.com"
// Dereference the 'h_addr_list' field twice to get the first element in the
// array, which is an array of 4 bytes representing the IPv4 address of the host.
// Roughly equivalent C:
// printf("%u.%u.%u.%u", (int) h_addr_list[0][0], (int) h_addr_list[0][1],
// (int) h_addr_list[0][2], (int) h_addr_list[0][3]);
google.contents.h_addr_list.contents.contents.toString();
===> "ctypes.uint8_t.array(4)([74, 125, 19, 17])" // 74.125.19.173. Creating C function pointers for JS functions.
// Declare the type of a comparator function that takes two pointers to elements,
// and returns:
// -1 if i < j;
// 0 if i == j;
// 1 if i > j.
// Equivalent C:
// typedef int (comparator_t*)(const int8_t* i, const int8_t* j);
let comparator_t = ctypes.FunctionType(ctypes.default_abi, ctypes.int,
ctypes.int8_t.ptr, ctypes.int8_t.ptr).ptr;
// What's the C type of 'comparator_t'?
comparator_t.name;
===> "int (*)(int8_t*,int8_t*)"
// Declare the 'qsort' function which takes an array of elements and a comparator
// function, and sorts the array.
// void qsort(void* array, size_t length, size_t elemsize, comparator_t comp);
let qsort = library.declare("qsort", ctypes.default_abi, ctypes.void_t,
ctypes.voidptr_t, ctypes.size_t, ctypes.size_t, comparator_t);
// Implement a JS function that looks just like 'comparator_t' above.
function reverse(i, j) { return j.contents - i.contents; }
// Construct a C function pointer from our JS function.
let reverse_ptr = comparator_t(reverse);
// What's 'reverse_ptr'?
reverse_ptr.toString();
===> "ctypes.FunctionType(ctypes.default_abi, ctypes.int, ctypes.int8_t.ptr,
ctypes.int8_t.ptr).ptr(ctypes.UInt64("0x81a430cb"))"
// Construct an array of values and call 'qsort'.
let array_t = ctypes.int8_t.array();
let ints = array_t([3, 1, 5, 6, 4, 2]);
qsort(ints.address(), ints.length, array_t.elementType.size, reverse_ptr);
// Voilà!
ints.toString();
===> "ctypes.int8_t.array(6)([6, 5, 4, 3, 2, 1])"So, if you’re an extension author, or writing code for the browser, keep js-ctypes in mind — and let us know how it goes!
Edit 8/7/10: updated syntax for API changes.
Makes me wonder if we should just rip out the awesomely painful xpconnect file io stuff and use ctyped native bindings
Comment by tglek — March 12, 2010 @ 8:45 pm
I really don’t know anything about Gecko architecture, so this may be a very silly question, but:
Could you strip out XPCOM from various components and just use js-ctypes to access those components from javascript? Would this improve performance or code readability, etc.?
Comment by voracity — March 12, 2010 @ 9:41 pm
Cool, I think I might take a shot at my first addon by writing something to hook to avahi to do the dns-sd bookmarks thing.
Comment by Spudd86 — March 12, 2010 @ 10:13 pm
tglek: in time, if the result is clearer, simpler (and therefore less buggy) code, why not? But beware that Minefield is currently at version number 3.7a3pre, and has been for some time, so I guess the “next version” of your extension (or the one after that) will have to drop support of this and earlier Firefox versions (i.e. have a “later” minVersion) if it wants to use “only” js-ctype bindings, without even falling back on the older stuff if ctypes.jsm cannot be imported.
dwitte: You know, of course (but maybe some lurkers don’t), that at step 1.2, “libc.so.6″ is typical of Linux; the cross-platform code would here have to open some DLL for Windows, or (IIRC) something else again for the Mac. Hopefully the rest is cross-plaform. (Maybe some “helper function” can be added to ctypes.jsm, possibly making use of some appropriate platform-dependent C module(s), for “open the C library for the current platform” — something which will be needed by practically anything using ctypes.jsm anyway?)
Comment by Tony Mechelynck — March 12, 2010 @ 10:44 pm
ctypes is available on firefox 3.6, afaics from the configure script and source code. Now, the big problem with ctypes is that there are more chances people will implement win32-only extensions…
Comment by glandium — March 13, 2010 @ 12:37 am
Does ctypes.open() cause synchronous I/O? Also, how would calling into Objective C code look like?
Comment by Markus Stange — March 13, 2010 @ 3:50 am
Interesting. Similar functionality is in Cycript (currently based on WebKit, and used by a bunch of us iPhone hackers to spelunk the frameworks), but based around Objective-C type signature strings rather than giant enums. Cycript also has a preprocessor that let’s you use the */-> operators for pointer indirection. (The preprocessor can be used alone as a separate compile step, and may be valuable to people using ctypes here, by just doing a search/replace for “$cyi” to “contents” in the resulting JavaScript).
One thing I’m curious about (as I hadn’t yet found a good solution for this): how is the closure generated for reverse_pre being managed? If I hand that pointer off to some C code (which will store it and call it at a later time) and never use the pointer again, even if I still try to hold a reference to it, might not an advanced GC (this could even be simple, such as in the case of a tail call) still determine the value is collectable? In which case, the memory management for those closures has to be manually handled by the developer. Is there a better way of thinking about this problem? Are JavaScript GC’s in fact required to not be that awesome?
Comment by Jay Freeman (saurik) — March 13, 2010 @ 1:09 pm
For those on OS X, you can try this out by just replacing “libc.so.6″ with “libc.dylib”.
But the general case of having a platform-independent way of opening the library is tracked in bug 518130.
https://bugzilla.mozilla.org/show_bug.cgi?id=518130
This would perhaps let you write something like ctypes.open(“libc”) without having to specify the extension.
Comment by Edward Lee — March 13, 2010 @ 2:01 pm
Social comments and analytics for this post…
This post was mentioned on Twitter by PlanetRepeater: Dan Witte: Extension authors, browser hackers, meet js-ctypes http://dlvr.it/BnKx...
Trackback by Social comments and analytics for this post... — March 14, 2010 @ 3:42 am
@tglek and @voracity — I’m all for people pushing the envelope.
There’s a bug on file for giving ctypes the ability to call IDL interface methods; see https://bugzilla.mozilla.org/show_bug.cgi?id=505907. It would improve performance for sure, but we need numbers here. I doubt it’d help readability, since XPConnect already makes calling IDL methods pretty easy.
@Tony — Yes, the example code would need a switch statement to pick the right libname depending on platform. This is currently possible in a hacky way; as Mardak says, see https://bugzilla.mozilla.org/show_bug.cgi?id=518130 for a better solution.
@glandium — ctypes is available in Firefox 3.6, but only provides basic integer and string (8-bit and 16-bit character) types. No pointers, arrays, structs, or function pointers.
@Markus — Yeah, library loading (and symbol resolution) will be synchronous. We could add an async API for it without much trouble, I think. No idea about Objective C.
@Jay — Lifetime of objects (pointers, in particular, like you say) are the responsibility of the JS programmer to manage. In practice, this just means that if you want a function pointer to live for the life of the app, stash it in a global variable. And make sure your script stays alive for a sufficiently long time. In Firefox, the event queue makes this easy. It’s unfortunate that the JS programmer has to worry about lifetime, but there’s really no way for us to manage it for you — there’s no way of knowing what the C API you’re interacting with expects.
Comment by dwitte — March 15, 2010 @ 5:09 pm
[...] extensions. ctypes.jsm now supports complex types including structures, pointers, and arrays. See Dan Witte’s post for more [...]
Pingback by n'1fo[r-matik] » Mozilla Firefox 3.7 Alpha 3 — March 17, 2010 @ 4:06 pm
[...] [...]
Pingback by Dritte Alpha von Firefox 3.7 erschienen - Software | News | ZDNet.de — March 18, 2010 @ 7:05 am
[...] extensions. ctypes.jsm now supports complex types including structures, pointers, and arrays. See Dan Witte’s post for more [...]
Pingback by Firefox 3.7 Alpha 3 Released | Dotnetwizard.net — March 21, 2010 @ 2:26 am
[...] JS-ctypes, our new easy-to-use system for extension authors who want to call into native code now has support for complex types: structures, pointers, and arrays. For more information on this, and how easy it can make calling into native code from JavaScript, see Dan Witte’s post. [...]
Pingback by Mozilla Developer Preview 4 Ready for Testing ✩ Mozilla Hacks – the Web developer blog — April 12, 2010 @ 11:41 am
[...] JS-ctypes, our new easy-to-use system for extension authors who want to call into native code now has support for complex types: structures, pointers, and arrays. For more information on this, and how easy it can make calling into native code from JavaScript, see Dan Witte’s post. [...]
Pingback by Mozilla Developer Preview (Number 4) Now Available | Easy Firefox — April 13, 2010 @ 12:00 am