Parsing JSON Really Quickly: Lessons Learned

This is an amazing presentation. Thank you!

GTA5 uses the inverse of this for its json parsing, it takes 6 minutes to parse a 10 megabyte json file for some ungodly reason

Now i wonder what magic a binary format could do with SIMD bit magic. I know Protocol_Buffer and some other like BJSON are not doing this.

Very impressive, and nice speech. Thanks.

Great example, thank you. DeltaJSON is really useful if you are working with JSON, it does compare, merge and graft.

If anyone else is confused by slide 40, 41; then there is an error where the order should be brackets -> 1, comma ->2, colon ->4, most whitespace -> 8, whitespace -> 16, other -> 0. Then the math checks out. Had me scratching my head for a long time...

If you want fast float serialization you should probably dump the in-memory representation, no?

"Our backend developers spend half of their time serializing and deserializing json."
From what i've been working around, JSON serializing and deserializing happens and only happens in the entrance or the exit of a request process. During the request process the "JSON" will be something like map/dict or Object. And in a typical backend server, the time saved by JSON serializing compares to DB calls will be almost negligible. But the magical branch-less parsing is so amazing and beautiful.

Now I wish I had a sticker!

Its really excellent approach. But I have no idea about how to check the cycles.. Is there any tool?

If you have learned to program assembly language, all of these optimizations are second nature. Everyone should learn an assembly language and spend time hand optimizing their code. Learning assembly will make you a better programmer and influence your coding strategies in time-critical areas. That being said, optimizations often cause readability to suffer, so be wise about how you use coding strategies.

I think if you need that level of performance, just ditch JSON in favor of something binary based like Protobuf.

Good old bitwise tricks. Reminds me how I coded 30 years ago.

Or... stop using JSON and start using Protocol Buffers. Better in virtually every way.

You sir, confirmed a theory I've had for the past year but was unable to work out: you absolutely can parse (at least partially) using SIMD. Which opens up the door to parsing with a GPU. Absolutely fantastic!

Getting rid of JSON might be a good idea then.

This is a great talk. Really clever optimizations.


However, it reminds me that we've fallen into a pit of failure because javascript is the lingua franca of the web. JSON is the fault of javascript. Easy to parse on the client because it can simply be "eval"ed to parse in the browser. Messages back to the server also require parsing. And the key here is that it requires *parsing*. A binary format, such as protocol buffers, is much more efficient for data transfer, both from a payload size and processing cost perspective, because it isn't *parsed*, per se, it is simply *read*. The problem is that javascript wasn't (isn't?) really capable of dealing with binary payloads, so binary solutions weren't, historically, an option.


Anyway, it makes me hope that the advent of WebAssembly will offer better solutions in the next few years. It is silly to me that great minds are wasting time optimizing things at this layer of abstraction, when the abstraction itself (JSON) is flawed.

👌🏼

Zen2 also has AVX512

Very nice talk.

I love this! I never knew about carry-less multiplication. I also didn’t realize that disk speeds and CPU throughput were so similar nowadays

And the moral of the story is: don't use JSON

Impressive bitfields wizardry 😮 Thank you. This talk is giving me inspiration to rewrite a few things without branches, and learn more about "bitwise" intrinsic operators !

Nice content, but somewhat painful to listen to.

Aren't you supposed to use classes like LeftSquareBracketMatcherStrategyFactory?

You might be faster by not using the jvm which needs todo emulations of the real platform e.g use posix write/read instead of java‘s sh… and move to a language that was actually designed to be easier analyzed and optimized.

Interesting lecture with good mix of high-level strategy and low-level (pseudo) implementation details.

I have started learning Jason this year can somebody tell me what's going on on the video

Good lecture for programmers interested in thinking pragmatically rather than just abstractly.