Skip to content

feat: add scyjava-stubgen cli command, and scyjava.types namespace, which provide type-safe imports with lazy init #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

tlambert03
Copy link
Collaborator

many more details and tests to follow... but just wanted to open this as a WIP.

Basic idea, after checking out this branch and running pip install -e . again:

  1. create stubs with a cli commend, e.g. scyjava-stubgen org.scijava:parsington:3.1.0
  2. Import names provided by that endpoint: python -c "from scyjava.types.org.scijava.parsington import Function; print(Function(1))". Only at the moment of class instantiation will the jvm be started.

Copy link

codecov bot commented Apr 25, 2025

Codecov Report

Attention: Patch coverage is 77.46479% with 48 lines in your changes missing coverage. Please review.

Project coverage is 75.98%. Comparing base (34bfae2) to head (0d231cc).
Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
src/scyjava/_stubs/_cli.py 49.20% 32 Missing ⚠️
src/scyjava/_stubs/_genstubs.py 89.15% 9 Missing ⚠️
src/scyjava/_stubs/_dynamic_import.py 88.88% 4 Missing ⚠️
tests/test_stubgen.py 92.59% 2 Missing ⚠️
src/scyjava/_jvm.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main      #82       +/-   ##
===========================================
+ Coverage   52.72%   75.98%   +23.25%     
===========================================
  Files          12       20        +8     
  Lines        1303     1653      +350     
===========================================
+ Hits          687     1256      +569     
+ Misses        616      397      -219     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tlambert03
Copy link
Collaborator Author

tlambert03 commented May 1, 2025

hey @ctrueden, I'm struggling to run the jep tests locally (on my mac). I have openjdk version "11.0.27" 2025-04-15 but still get:

--> tests/it/java_heap.py [OK]
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

Traceback (most recent call last):
  File "/Users/talley/dev/self/scyjava/tests/it/jvm_version.py", line 11, in <module>
    before_version = scyjava.jvm_version()
  File "/Users/talley/dev/self/scyjava/src/scyjava/_jvm.py", line 70, in jvm_version
    default_jvm_path = jpype.getDefaultJVMPath()
  File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 70, in getDefaultJVMPath
    return finder.get_jvm_path()
           ~~~~~~~~~~~~~~~~~~~^^
  File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 184, in get_jvm_path
    jvm = method()
  File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 311, in _javahome_binary
    return subprocess.check_output(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        ['/usr/libexec/java_home']).strip()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 472, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               **kwargs).stdout
               ^^^^^^^^^
  File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/libexec/java_home']' returned non-zero exit status 1.

it's a bit unclear to me whether I should be trying to get these to work at all with jep, or just add a skip to the test

@ctrueden
Copy link
Member

ctrueden commented May 1, 2025

As a workaround, you could try setting JAVA_HOME? Then maybe jpype wouldn't try to invoke the java_home command. Out of curiosity: what does /usr/libexec/java_home -V say when you run it from the CLI?

@tlambert03
Copy link
Collaborator Author

got that working... and commented out the "don't run on macos cause it's flaky" bit, and now get this:

-------------------------------------------
|  Testing Jep mode (Python inside Java)  |
-------------------------------------------
DEBUG 2025-05-01 08:20:23,154: Using settings:     {'m2repo': '/Users/talley/.m2/repository', 'cachedir': '/Users/talley/.jgo', 'links': 'auto'}
DEBUG 2025-05-01 08:20:23,155: Using repositories: {'scijava.public': 'https://maven.scijava.org/content/groups/public'}
DEBUG 2025-05-01 08:20:23,155: Using shortcuts:    {}
DEBUG 2025-05-01 08:20:23,155: Returning expanded coordinate black.ninia:jep:jep.Run.
DEBUG 2025-05-01 08:20:23,155: Returning expanded coordinate org.scijava:scijava-table.
INFO 2025-05-01 08:20:23,155: First time start-up may be slow. Downloaded dependencies will be cached for shorter start-up times in subsequent executions.
DEBUG 2025-05-01 08:20:23,155: Executing: ('/Users/talley/Library/Caches/cjdk/v0/misc-dirs/98cdba9371f93e1b5b8b95941b562d1647aecc21/apache-maven-3.9.9/bin/mvn', '-B', '-f', '/Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/pom.xml', 'dependency:resolve', '-X')
DEBUG 2025-05-01 08:20:31,600: Relevant mvn output: [INFO]    black.ninia:jep:jar:4.2.2:compile -- module jep (auto)
DEBUG 2025-05-01 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type auto
DEBUG 2025-05-01 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.scijava:scijava-table:jar:1.0.2:compile -- module org.scijava.table [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.scijava:scijava-common:jar:2.89.0:compile -- module org.scijava [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.scijava:parsington:jar:3.0.0:compile -- module org.scijava.parsington [auto]
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type hard
DEBUG 2025-05-01 08:20:31,601: Relevant mvn output: [INFO]    org.bushe:eventbus:jar:1.4:compile -- module eventbus (auto)
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type auto
DEBUG 2025-05-01 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type hard
DEBUG 2025-05-01 08:20:31,602: Relevant mvn output: [INFO]    org.scijava:scijava-optional:jar:1.0.1:compile -- module org.scijava.optional [auto]
DEBUG 2025-05-01 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type auto
DEBUG 2025-05-01 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type hard
DEBUG 2025-05-01 08:20:31,602: class path: /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/*
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001716a3000 (most recent call first):
  <no Python frame>

that's probably not the flaky bit right? looks like a poor setup of the python environment

@tlambert03
Copy link
Collaborator Author

i think i see the issue, i'll work on the jgo ... jep_test.py command directly and see if I can figure out what assumptions it's making about my environment setup

@tlambert03
Copy link
Collaborator Author

after digging a bit deeper into how jgo and jep itself is working, I'm skipping stubgen tests on jep for now. it seems extremely dependent on some careful manual setup of the python environment. I was able to get it to find my standard library, but then it lost the pure python parts of jep. After fixing that, it was unable to find my (editable) install of scyjava because it's not following .pth files in a standard way. I don't understand all the variables well enough yet (I don't understand who's doing the magic, whether it's jep.Run itself or the jgo command line), so don't know where to attack it

@tlambert03
Copy link
Collaborator Author

code-wise, this is ready to go... however, I don't expect it to be "human-interpretable" yet. There are many ways we could choose to document this and encourage/discourage its usage in various scenarios. Might be best to have a zoom about it so we can tinker with it together and discuss

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants