read_json: ValueError: Value is too big

[jmh045000]

Reopening issue #14530. The close description is incorrect. The JSON specification explicitly states that limits are not in the specification.

From https://tools.ietf.org/html/rfc7159#section-6

This specification allows implementations to set limits on the range and precision of numbers accepted

The standard json library in python supports large numbers, meaning the language supports JSON with these values.

Python 3.6.8 (default, Apr  7 2019, 21:09:51)
[GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> j = """{"id": 9253674967913938907}"""
>>> json.loads(j)
{'id': 9253674967913938907}

Loading a json file with large integers (> 2^32), results in "Value is too big". I have tried changing the orient to "records" and also passing in dtype={'id': numpy.dtype('uint64')}. The error is the same.

import pandas
data = pandas.read_json('''{"id": 10254939386542155531}''')
print(data.describe())

Expected Output

                          id
count                      1
unique                     1
top     10254939386542155531
freq                       1

Actual Output (even with dtype passed in)

 File "./parse_dispatch_table.py", line 34, in <module>
    print(pandas.read_json('''{"id": 10254939386542155531}''', dtype=dtype_conversions).describe())
  File "/users/XXX/.local/lib/python3.4/site-packages/pandas/io/json.py", line 234, in read_json
    date_unit).parse()
  File "/users/XXX/.local/lib/python3.4/site-packages/pandas/io/json.py", line 302, in parse
    self._parse_no_numpy()
  File "/users/XXX/.local/lib/python3.4/site-packages/pandas/io/json.py", line 519, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Value is too big

No problem using read_csv:

import pandas
import io
print(pandas.read_csv(io.StringIO('''id\n10254939386542155531''')).describe())

Output using read_csv

                          id
count                      1
unique                     1
top     10254939386542155531
freq                       1

Output of pd.show_versions()

[artdgn]

Workaround:

A workaround that worked for me is to patch the loads function that's used in read_json.
You can patch it using the default python json module or using simplejson instead of the standard python json module.
E.g. if you monkeypatch pandas like this:

import pandas.io.json

# monkeypatch using standard python json module
import json
pd.io.json._json.loads = lambda s, *a, **kw: json.loads(s)

# monkeypatch using faster simplejson module
import simplejson
pd.io.json._json.loads = lambda s, *a, **kw: simplejson.loads(s)

# normalising (unnesting) at the same time (for nested jsons)
pd.io.json._json.loads = lambda s, *a, **kw: pandas.io.json.json_normalize(simplejson.loads(s))

After this patch read_json() should work on the example above (pandas.read_json('''{"id": 10254939386542155531}''', orient='index') and on the JSON that was giving me trouble.

Suggestion for future solution:

A possible solution for pandas is to provide a parameter via which one can override (inject) the loads function when calling read_json(). E.g read_json(data, *args, loads_fn=None, **kwargs). If the maintainers think that's a viable solution, I'd be happy to submit a PR.

[mroeschke]

Going to close this issue as I believe it's the same issue as #20599

[deponovo]

I don't think this issue is solved.

Test data (test.csv:):

data1,data2
10254939386542155531,12764512476851234

Test code:

import pandas as pd
import json

df = pd.read_csv(r'test.csv')  # all fine
df
Out[45]: 
                  data1              data2
0  10254939386542155531  12764512476851234

df.dtypes
Out[46]: 
data1    uint64
data2     int64
dtype: object

pd.read_json(json.dumps(df.to_dict()))  # prob

Results in error:

  File "<env_path>\lib\site-packages\pandas\io\json\_json.py", line 1140, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None
ValueError: Value is too big

then:

json.loads(json.dumps(df.to_dict()))
Out[48]: {'data1': {'0': 10254939386542155531}, 'data2': {'0': 12764512476851234}}

This behavior is therefore inconsistent.
Also, 10254939386542155531 is allowed as uint64.

(Pandas 1.3.4, json 2.0.9, numpy 1.21.3)

[jreback]

@deponovo this was redirected to another open issue - look there and see if the example is sufficient

issue get fixed when the members of the community puts up pull requests

the core team can provide code review

[deponovo]

Is there a wiki how I could do this? Would be my first time.

[deponovo]

take

[krpatter-intc]

I'm using pandas 1.4.1 and can still reproduce this issue:

pandas.read_json(pandas.DataFrame([{"foo":441350462101887021235463396661461057}]).to_json(orient="table"), orient="table")

Sorry one thing to note: it seems the output (to_json) knows this is a problem and puts the type as string, but still writes to the json as an python long (instead of a string):

'{"schema":{"fields":[{"name":"index","type":"integer"},{"name":"foo","type":"string"}],"primaryKey":["index"],"pandas_version":"1.4.0"},"data":[{"index":0,"foo":441350462101887021235463396661461057}]}'

[krpatter-intc]

@deponovo Would it be helpful to open a new issue for this?

Is it ok to do the json overrides I see in other related issues, is it a performance trade off and that is why pandas uses a custom json parser?

[dmitriyshashkin]

Workaround posted above doesn't seem to work anymore. Does anyone have a new workaround that works?