Logo

dev-resources.site

for different kinds of informations.

iter.json: A Powerful and Efficient Way to Iterate and Manipulate JSON in Go

Published at
12/13/2024
Categories
go
iterator
json
development
Author
olvrng
Main Article
https://dev.to/olvrng/iterjson-a-powerful-and-efficient-way-to-iterate-and-manipulate-json-in-go-12cl
Categories
4 categories in total
go
open
iterator
open
json
open
development
open
Author
6 person written this
olvrng
open
iter.json: A Powerful and Efficient Way to Iterate and Manipulate JSON in Go

Have you ever needed to modify unstructured JSON data in Go? Maybe you’ve had to delete all blacklisted fields, rename keys from camelCase to snake_case, or convert all number ids to strings because JavaScript does not like int64? If your solution has been to unmarshal everything into a map[string]any using encoding/json and then marshal it back... well, let’s face it, that’s far from efficient!

What if you could loop through the JSON data, grab the path of each item, and decide exactly what to do with it on the fly?

Yes! I have a good news! With the new iterator feature in Go 1.23, there’s a better way to iterate and manipulate JSON. Meet ezpkg.io/iter.json — your powerful and efficient companion for working with JSON in Go.

1. Iterating JSON

Given that we have an alice.json file:

{
  "name": "Alice",
  "age": 24,
  "scores": [9, 10, 8],
  "address": {
    "city": "The Sun",
    "zip": 10101
  }
}
Enter fullscreen mode Exit fullscreen mode

First, let's use for range Parse() to iterate over the JSON file, then print the path, key, token, and level of each item. See examples/01.iter.

package main

import (
    "fmt"

    "ezpkg.io/errorz"
    iterjson "ezpkg.io/iter.json"
)

func main() {
    data := `{"name": "Alice", "age": 24, "scores": [9, 10, 8], "address": {"city": "The Sun", "zip": 10101}}`

    // 🎄Example: iterate over json
    fmt.Printf("| %12v | %10v | %10v |%v|\n", "PATH", "KEY", "TOKEN", "LVL")
    fmt.Println("| ------------ | ---------- | ---------- | - |")
    for item, err := range iterjson.Parse([]byte(data)) {
        errorz.MustZ(err)

        fmt.Printf("| %12v | %10v | %10v | %v |\n", item.GetPathString(), item.Key, item.Token, item.Level)
    }
}
Enter fullscreen mode Exit fullscreen mode

The code will output:

|         PATH |        KEY |      TOKEN |LVL|
| ------------ | ---------- | ---------- | - |
|              |            |          { | 0 |
|         name |     "name" |    "Alice" | 1 |
|          age |      "age" |         24 | 1 |
|       scores |   "scores" |          [ | 1 |
|     scores.0 |            |          9 | 2 |
|     scores.1 |            |         10 | 2 |
|     scores.2 |            |          8 | 2 |
|       scores |            |          ] | 1 |
|      address |  "address" |          { | 1 |
| address.city |     "city" |  "The Sun" | 2 |
|  address.zip |      "zip" |      10101 | 2 |
|      address |            |          } | 1 |
|              |            |          } | 0 |
Enter fullscreen mode Exit fullscreen mode

2. Building JSON

Use Builder to build a JSON data. It accepts optional arguments for indentation. See examples/02.builder.

b := iterjson.NewBuilder("", "    ")
// open an object
b.Add("", iterjson.TokenObjectOpen)

// add a few fields
b.Add("name", "Alice")
b.Add("age", 22)
b.Add("email", "[email protected]")
b.Add("phone", "(+84) 123-456-789")

// open an array
b.Add("languages", iterjson.TokenArrayOpen)
b.Add("", "English")
b.Add("", "Vietnamese")
b.Add("", iterjson.TokenArrayClose)
// close the array

// accept any type that can marshal to json
b.Add("address", Address{
    HouseNumber: 42,
    Street:      "Ly Thuong Kiet",
    City:        "Ha Noi",
    Country:     "Vietnam",
})

// accept []byte as raw json
b.Add("pets", []byte(`[{"type":"cat","name":"Kitty","age":2},{"type":"dog","name":"Yummy","age":3}]`))

// close the object
b.Add("", iterjson.TokenObjectClose)

out := errorz.Must(b.Bytes())
fmt.Printf("\n--- build json ---\n%s\n", out)
Enter fullscreen mode Exit fullscreen mode

Which will output the JSON with indentation:

{
    "name": "Alice",
    "age": 22,
    "email": "[email protected]",
    "phone": "(+84) 123-456-789",
    "languages": [
        "English",
        "Vietnamese"
    ],
    "address": {"house_number":42,"street":"Ly Thuong Kiet","city":"Ha Noi","country":"Vietnam"},
    "pets": [
        {
            "type": "cat",
            "name": "Kitty",
            "age": 2
        },
        {
            "type": "dog",
            "name": "Yummy",
            "age": 3
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

3. Formatting JSON

You can reconstruct or format a JSON data by sending its key and values to a Builder. See examples/03.reformat.

{
    // 🐝Example: minify json
    b := iterjson.NewBuilder("", "")
    for item, err := range iterjson.Parse(data) {
        errorz.MustZ(err)
        b.AddRaw(item.Key, item.Token)
    }
    out := errorz.Must(b.Bytes())
    fmt.Printf("\n--- minify ---\n%s\n----------\n", out)
}
{
    // 🦋Example: format json
    b := iterjson.NewBuilder("👉   ", "\t")
    for item, err := range iterjson.Parse(data) {
        errorz.MustZ(err)
        b.AddRaw(item.Key, item.Token)
    }
    out := errorz.Must(b.Bytes())
    fmt.Printf("\n--- reformat ---\n%s\n----------\n", out)
}
Enter fullscreen mode Exit fullscreen mode

The first example minifies the JSON while the second example formats it with prefix "👉" on each line.

--- minify ---
{"name":"Alice","age":24,"scores":[9,10,8],"address":{"city":"The Sun","zip":10101}}
----------

--- reformat ---
👉   {
👉       "name": "Alice",
👉       "age": 24,
👉       "scores": [
👉           9,
👉           10,
👉           8
👉       ],
👉       "address": {
👉           "city": "The Sun",
👉           "zip": 10101
👉       }
👉   }
----------
Enter fullscreen mode Exit fullscreen mode

4. Adding line numbers

In this example, we add line numbers to the JSON output, by adding a b.WriteNewline() before the fmt.Fprintf() call. See examples/04.line_number.

// 🐞Example: print with line number
i := 0
b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)
    b.WriteNewline(item.Token.Type())

    // 👉 add line number
    fmt.Fprintf(b, "%3d    ", i)
    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- line number ---\n%s\n----------\n", out)
Enter fullscreen mode Exit fullscreen mode

This will output:

  1    {
  2        "name": "Alice",
  3        "age": 24,
  4        "scores": [
  5            9,
  6            10,
  7            8
  8        ],
  9        "address": {
 10            "city": "The Sun",
 11            "zip": 10101
 12        }
 13    }
Enter fullscreen mode Exit fullscreen mode

5. Adding comments

By putting a fmt.Fprintf(comment) between b.WriteComma() and b.WriteNewline(), you can add a comment to the end of each line. See examples/05.comment.

i, newlineIdx, maxIdx := 0, 0, 30
b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    b.WriteComma(item.Token.Type())

    // 👉 add comment
    if i > 0 {
        length := b.Len() - newlineIdx
        fmt.Fprint(b, strings.Repeat(" ", maxIdx-length))
        fmt.Fprintf(b, "// %2d", i)
    }
    i++

    b.WriteNewline(item.Token.Type())
    newlineIdx = b.Len() // save the newline index

    b.Add(item.Key, item.Token)
}
length := b.Len() - newlineIdx
fmt.Fprint(b, strings.Repeat(" ", maxIdx-length))
fmt.Fprintf(b, "// %2d", i)

out := errorz.Must(b.Bytes())
fmt.Printf("\n--- comment ---\n%s\n----------\n", out)
Enter fullscreen mode Exit fullscreen mode

This will output:

{                             //  1
    "name": "Alice",          //  2
    "age": 24,                //  3
    "scores": [               //  4
        9,                    //  5
        10,                   //  6
        8                     //  7
    ],                        //  8
    "address": {              //  9
        "city": "The Sun",    // 10
        "zip": 10101          // 11
    }                         // 12
}                             // 13
Enter fullscreen mode Exit fullscreen mode

6. Filtering JSON and extracting values

There are item.GetPathString() and item.GetRawPath() to get the path of the current item. You can use them to filter the JSON data. See examples/06.filter_print.

Example with item.GetPathString() and regexp:

fmt.Printf("\n--- filter: GetPathString() ---\n")
i := 0
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)

    path := item.GetPathString()
    switch {
    case path == "name",
        strings.Contains(path, "address"):
        // continue
    default:
        continue
    }

    // 👉 print with line number
    fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath())
}
Enter fullscreen mode Exit fullscreen mode

Example with item.GetRawPath() and path.Match():

fmt.Printf("\n--- filter: GetRawPath() ---\n")
i := 0
for item, err := range iterjson.Parse(data) {
    i++
    errorz.MustZ(err)

    path := item.GetRawPath()
    switch {
    case path.Match("name"),
        path.Contains("address"):
        // continue
    default:
        continue
    }

    // 👉 print with line number
    fmt.Printf("%2d %20s . %s\n", i, item.Token, item.GetPath())
}
Enter fullscreen mode Exit fullscreen mode

Both examples will output:

 2              "Alice" . name
 9                    { . address
10            "The Sun" . address.city
11                10101 . address.zip
12                    } . address
Enter fullscreen mode Exit fullscreen mode

7. Filtering JSON and returning a new JSON

By combining the Builder with the option SetSkipEmptyStructures(false) and the filtering logic, you can filter the JSON data and return a new JSON. See examples/07.filter_json

// 🦁Example: filter and output json
b := iterjson.NewBuilder("", "    ")
b.SetSkipEmptyStructures(true) // 👉 skip empty [] or {}
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    if item.Token.IsOpen() || item.Token.IsClose() {
        b.Add(item.Key, item.Token)
        continue
    }

    path := item.GetPathString()
    switch {
    case path == "name",
        strings.Contains(path, "address"):
        // continue
    default:
        continue
    }

    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- filter: output json ---\n%s\n----------\n", out)
Enter fullscreen mode Exit fullscreen mode

This example will return a new JSON with only the filtered fields:

{
    "name": "Alice",
    "address": {
        "city": "The Sun",
        "zip": 10101
    }
}
Enter fullscreen mode Exit fullscreen mode

8. Editing values

This is an example for editing values in a JSON data. Assume that we are using number ids for our API. The ids are too big and JavaScript can't handle them. We need to convert them to strings. See examples/08.number_id and order.json.

Iterate over the JSON data, find all _id fields and convert the number ids to strings:

b := iterjson.NewBuilder("", "    ")
for item, err := range iterjson.Parse(data) {
    errorz.MustZ(err)
    key, _ := item.GetRawPath().Last().ObjectKey()
    if strings.HasSuffix(key, "_id") {
        id, err0 := item.Token.GetInt()
        if err0 == nil {
            b.Add(item.Key, strconv.Itoa(id))
            continue
        }
    }
    b.Add(item.Key, item.Token)
}
out := errorz.Must(b.Bytes())
fmt.Printf("\n--- convert number id ---\n%s\n----------\n", out)
Enter fullscreen mode Exit fullscreen mode

This will add quotes to the number ids:

{
    "order_id": "12345678901234",
    "number": 12,
    "customer_id": "12345678905678",
    "items": [
        {
            "item_id": "12345678901042",
            "quantity": 1,
            "price": 123.45
        },
        {
            "item_id": "12345678901098",
            "quantity": 2,
            "price": 234.56
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Conclusion

The ezpkg.io/iter.json package empowers Go developers to handle JSON data with precision and efficiency. Whether you need to iterate through complex JSON structures, build new JSON objects dynamically, format or minify data, filter specific fields, or even transform values, iter.json offers a flexible and powerful solution.

I’m excited to share this package with the community as a tool for effective JSON manipulation without the need for fully parsing the data. While it’s still in early development and there’s room for more features, it already works well for many common use cases.

If you have specific requirements or ideas for improvement, feel free to reach out — I’d love to hear your feedback and help support your use cases! 🥳

Author

I'm Oliver Nguyen .  A software engineer working with Go and JS. I enjoy learning and seeing a better version of myself each day. Occasionally spin off new open source projects. Share knowledge and thoughts during my journey.

The post is also published at olivernguyen.io.

json Article's
30 articles in total
Favicon
How to Fetch URL Content, Set It into a Dictionary, and Extract Specific Keys in iOS Shortcuts
Favicon
Dynamic Routes in Astro (+load parameters from JSON)
Favicon
Effortlessly Host Static JSON Files with JSONsilo.com
Favicon
How to Implement Authentication in React Using JWT (JSON Web Tokens)
Favicon
Converting documents for LLM processing — A modern approach
Favicon
Import JSON Data in Astro (with Typescript)
Favicon
Devise not accepting JSON Token
Favicon
Integration for FormatJS/react-intl: Automated Translations with doloc
Favicon
“Defu” usage in unbuild source code.
Favicon
Converting documents for LLM processing — A modern approach
Favicon
How to crawl and parse JSON data with Python crawler
Favicon
JSON Visual Edit
Favicon
Develop a ulauncher extension with a command database
Favicon
Building a Smart Feedback Agent with Copilot Studio, Adaptive cards and Power Automate
Favicon
Simplifying JSON Validation with Ajv (Another JSON Validator)
Favicon
A Straightforward Guide to Building and Using a JSON Database File
Favicon
AI prompt sample - a full chat content that demonstrates how to get a professional looking website in a few munities
Favicon
Fixing and Validating JSON with Ease: An In-Depth Guide
Favicon
Useful too to work with your JSON files - jq
Favicon
what is jq? a program for json files
Favicon
Code. Gleam. Extract fields from JSON
Favicon
Build an Application Without SQL Server Database (Avoiding RPrometheusedis, MongoDB, and )
Favicon
FAQ — Bloomer Mock Data Generator
Favicon
My React Journey: Day 18
Favicon
Working with JSON in MySQL
Favicon
JSON for Biggners
Favicon
angular and json
Favicon
iter.json: A Powerful and Efficient Way to Iterate and Manipulate JSON in Go
Favicon
This unknown Currency API is served over 50 Billion times a month !
Favicon
Common Data Formats in JavaScript: A Comprehensive Guide With Examples

Featured ones:

Favicon
Open
abubakersiddique761
Favicon
Open
0x3d_site
Favicon
Open
0x2e_tech
Favicon
Open
0x4c-quest
Favicon
Open
gittech