Commit graph

52 commits

Author SHA1 Message Date
Aaron Patterson
89d89fa49d When reading from stdin, put a wrapper around the IO object
The purpose of this commit is to fix Bug #21188.  We need to detect when
stdin has run in to an EOF case.  Unfortunately we can't _call_ the eof
function on IO because it will block.

Here is a short script to demonstrate the issue:

```ruby
x = STDIN.gets
puts x
puts x.eof?
```

If you run the script, then type some characters (but _NOT_ a newline),
then hit Ctrl-D twice, it will print the input string.  Unfortunately,
calling `eof?` will try to read from STDIN again causing us to need a
3rd Ctrl-D to exit the program.

Before introducing the EOF callback to Prism, the input loop looked
kind of like this:

```ruby
loop do
  str = STDIN.gets
  process(str)

  if str.nil?
    p :DONE
  end
end
```

Which required 3 Ctrl-D to exit.  If we naively changed it to something
like this:

```ruby
loop do
  str = STDIN.gets
  process(str)

  if STDIN.eof?
    p :DONE
  end
end
```

It would still require 3 Ctrl-D because `eof?` would block.  In this
patch, we're wrapping the IO object, checking the buffer for a newline
and length, and then using that to simulate a non-blocking eof? method.

This commit wraps STDIN and emulates a non-blocking `eof` function.

[Bug #21188]
2025-08-04 12:34:33 -07:00
Earlopain
026079925c [ruby/prism] Do not use 0 to indicate the latest ruby version to parse
This makes it hard to do version checks against this value. The current version checks work because there are so few possible values at the moment.

As an example, PR 3337 introduces new syntax for ruby 3.5 and uses `PM_OPTIONS_VERSION_LATEST` as its version guard. Because what is considered the latest changes every year, it must later be changed to `parser->version == parser->version == PM_OPTIONS_VERSION_CRUBY_3_5 || parser->version == PM_OPTIONS_VERSION_LATEST`, with one extra version each year.

With this change, the PR can instead write `parser->version >= PM_OPTIONS_VERSION_CRUBY_3_5` which is self-explanatory
and works for future versions.

8318a113ca
2025-07-29 17:17:28 +00:00
Nobuyoshi Nakada
991cf2dd4d [ruby/prism] [DOC] Specify markdown mode to RDoc
12af4e144e
2025-05-29 04:45:58 +00:00
Kevin Newton
641f15b1c6 [ruby/prism] Mark Prism as ractor-safe
c02429765b
2025-03-19 21:11:57 +00:00
Kevin Newton
af76b7f4d9 [ruby/prism] Revert "Mark extension as Ractor-safe"
56eaf53732
2025-03-12 19:56:22 +00:00
Kevin Newton
242e99eb0f [ruby/prism] Mark extension as Ractor-safe
10e5431b38
2025-03-12 19:15:03 +00:00
Kevin Newton
51d3d6ac8c [ruby/prism] Support forwarding flags on scopes
When parent scopes around an eval are forwarding parameters (like
*, **, &, or ...) we need to know that information when we are in
the parser. As such, we need to support passing that information
into the scopes option. In order to do this, unfortunately we need
a bunch of changes.

The scopes option was previously an array of array of strings.
These corresponded to the names of the locals in the parent scopes.
We still support this, but now additionally support passing in a
Prism::Scope instance at each index in the array. This Prism::Scope
class holds both the names of the locals as well as an array of
forwarding parameter names (symbols corresponding to the forwarding
parameters). There is convenience function on the Prism module that
creates a Prism::Scope object using Prism.scope.

In JavaScript, we now additionally support an object much the same
as the Ruby side. In Java, we now have a ParsingOptions.Scope class
that holds that information. In the dump APIs, these objects in all
3 languages will add an additional byte for the forwarding flags in
the middle of the scopes serialization.

All of this is in service of properly parsing the following code:

```ruby
def foo(*) = eval("bar(*)")
```

21abb6b7c4
2025-01-14 20:31:38 +00:00
Kevin Newton
da93c9ae29 [ruby/prism] Refactor serializer
8ab2532f09
2025-01-14 15:32:41 +00:00
Kevin Newton
713f31872a [ruby/prism] Freeze AST option
To make it so that you can pass `freeze: true` to Prism parse
methods and get back a deeply-frozen AST that is Ractor-
shareable.

8e6a93b2d2
2025-01-14 15:32:39 +00:00
Kevin Newton
b79152fd22 [ruby/prism] Support 3.5 for version option
6b6aa05bfb
2025-01-11 19:09:05 -05:00
Benoit Daloze
6c123649cd [ruby/prism] Use RbConfig to locate libprism and headers when it is a default gem
* This is notably necessary on TruffleRuby, which is updating to Ruby 3.3 which introduces Prism as a default gem.
* Using the existing path is not an option as it would end up in truffleruby/lib/build/libprism.so and
  "truffleruby/lib/include/#{header}" which are not good places for such files.

5d16473e69
2024-11-12 14:20:39 +00:00
Kevin Newton
5f62522d5b [ruby/prism] Prism::StringQuery
Introduce StringQuery to provide methods to access some metadata
about the Ruby lexer.

d3f55b67b9
2024-10-11 19:34:57 +00:00
Kevin Newton
414a848cc6 [ruby/prism] Accept version shorthand like 3.4
098f1c4607
2024-09-24 13:21:36 +00:00
Benoit Daloze
ed4a55fc4d [ruby/prism] Accept all 3.3.x and 3.4.x Ruby versions for Prism.parse
a4fcd5339a
2024-09-24 12:24:19 +00:00
Kevin Newton
f515a1ab4b [ruby/prism] Introduce partial_script option
b28877fa4f
2024-09-20 15:42:12 +00:00
Kevin Newton
f85efc9748 [ruby/prism] Expose main_script in serialization API
0b527ca93f
2024-09-13 19:13:21 +00:00
Kevin Newton
38ba15beed [ruby/prism] Check errno for parsing directory
d68ea29d04
2024-09-12 13:43:04 -04:00
Kevin Newton
d827d32527 [ruby/prism] Provide ability to lock encoding while parsing
f7faedfb3f
2024-06-10 17:21:32 -04:00
Kevin Newton
72452f4387 [ruby/prism] Tests overhaul
6f886be0a4
2024-05-30 15:18:20 -04:00
Kevin Newton
f8b750370e [ruby/prism] Remove Debug module
4d8929ff6a
2024-05-24 17:19:38 +00:00
Koichi ITO
b181ba7400 [ruby/prism] Use version: 3.3.1 against Translation::Parser
Follow up https://github.com/ruby/prism/pull/2760.

This PR updates the `Translation::Parser` to use version 3.3.1 when the version 3.3 is specified.
The Parser gem is structured to support the latest patch versions, hence this aligns with Parser-compatible versioning.
As noted in https://github.com/ruby/prism/pull/2760, the behavior remains unchanged with this switch from 3.3.0 to 3.3.1.

efde09d318
2024-05-04 16:31:58 +00:00
Vinicius Stock
4fbb208185 [ruby/prism] Create specialized ASCIISource with asciionly optimizations
40993166a8
2024-05-03 18:10:21 +00:00
Kevin Newton
23be6599a2 [ruby/prism] Split parse result based on type
17194e096d
2024-04-19 19:25:32 +00:00
Kevin Newton
d186eb36a4 [ruby/prism] Add a reflection API for determining the fields of a node
f3f9950a74
2024-04-17 13:54:29 -04:00
Kevin Newton
fcc06fa82a [ruby/prism] CLI -x flag
2068e3c30a
2024-03-28 12:04:35 -04:00
Kevin Newton
2ab75bc444 [ruby/prism] Support offset
665f533373
2024-03-11 14:49:23 +00:00
Kevin Newton
ec159fc8ba [ruby/prism] Support parsing streams
efdc2b7222
2024-03-07 20:40:39 +00:00
Kevin Newton
50e999c56d [ruby/prism] Command line options as a bitset
369ffbd57e
2024-02-29 12:05:19 -05:00
Kevin Newton
cd8d1018bb [ruby/prism] Resync RBI and test it in CI
4ef4032774
2024-02-29 16:29:16 +00:00
Kevin Newton
3ca8b4aee0 [ruby/prism] Support -p, -n, -a, and -l command line options
959eb506ca
2024-02-27 04:22:39 +00:00
Benoit Daloze
8f17b3bd27 [ruby/prism] Avoid extra String copies in the FFI backend
* For Prism.parse_file the file contents would be read as native, then
  converted to a Ruby String, then converted to a native String for
  pm_serialize_parse().
* Refactor the logic to always use a pm_string for the source code and
  pass that to other native functions.

9002b3c47d
2024-02-15 20:25:35 +00:00
Kevin Newton
29d04bb0c4 [ruby/prism] Introduce version: "3.4.0"
This is effectively an alias for "latest" right now. In the future
it will change to be its own enum value.

2c86036022
2024-02-13 18:26:28 +00:00
eileencodes
936c0ab5e8 [ruby/prism] Implement file parsing error handling
This PR implements proper file parsing error handling. Previously
`file_options` would call `pm_string_mapped_init` which would print an
error from `perror`. However this wouldn't raise a proper Ruby error so
it was just a string output. I've done the following:

- Raise an error from `rb_syserr_fail` with the filepath in
`file_options`.
- No longer return `Qnil` if `file_options` returns false (because now
it will raise)
- Update `file_options` to return `static void` instead of `static
bool`.
- Update `file_options` and `profile_file` to check the type so when
passing `nil` we see a `TypeError`.
- Delete `perror` from `pm_string_mapped_init`
- Update `FFI` backend to raise appropriate errors when calling
`pm_string_mapped_init`.
- Add tests for `dump_file`, `lex_file`, `parse_file`,
`parse_file_comments`, `parse_lex_file`, and `parse_file_success?`
when a file doesn't exist and for `nil`.
- Updates the `bin/parse` script to no longer raise it's own
`ArgumentError` now that we raise a proper error.

Fixes: ruby/prism#2207

b2f7494ff5
2024-02-06 20:49:33 +00:00
Kevin Newton
e9f1324464 Sync to latest prism 2024-02-01 12:52:16 -05:00
Cameron Dutro
8cbba87ca8 [ruby/prism] Add parse options to JavaScript's parsePrism function
d7fe7c7ae7
2024-01-16 19:19:30 +00:00
Kevin Newton
23beceedb7 [ruby/prism] IndexTargetNode should always have ATTRIBUTE_WRITE
Because this is a user-facing change, we also need to deal with the
fact that CRuby 3.3.0 was just released.

In order to support workflows that want to parse exactly as CRuby
parses in a specific version, this PR introduces a new option to
the options struct that is "version". This allows you to specify
that you want "3.3.0" parsing.

I'm not sure if this is the correct solution. Another solution is
to just fork and keep around the old branch for security patches.
Or we could keep around a copy of the source files within this
repository as another directory and only update when necessary.
There are a lot of potential solutions here.

Because this change is so small and the check for it is so minimal,
I've decided to go with this enum. If this ends up entirely
cluttering the codebase with version checks, we'll come up with
another solution. But for now this works, so we're going to go in
this direction for a bit until we determine it's no longer working.

d8c7e6bd10
2024-01-02 18:51:18 +00:00
Kevin Newton
492c82cb41 [ruby/prism] Prism.parse_success?(source)
A lot of tools use Ripper/RubyVM::AbstractSyntaxTree to determine
if a source is valid. These tools both create an AST instead of
providing an API that will return a boolean only.

This new API only creates the C structs, but doesn't bother
reifying them into Ruby/the serialization API. Instead it only
returns true/false, which is significantly more efficient.

7014740118
2023-12-01 20:53:34 +00:00
Kevin Newton
c798943a4a [ruby/prism] Move DATA parsing into its own parse result field
42b60b6e95
2023-11-28 13:25:48 +00:00
Haldun Bayhantopcu
8966d06b96 [ruby/prism] Warning for ENDs in methods
(https://github.com/ruby/prism/pull/1899)

1b41c2d56c
2023-11-21 16:36:12 +00:00
Kevin Newton
f2ed7eaba0 [ruby/prism] Add character APIs for locations
(https://github.com/ruby/prism/pull/1809)

d493ccd093
2023-11-20 16:07:06 +00:00
Kevin Newton
ed75518192
[ruby/prism] Rename librubyparser to libprism
librubyparser was an artifact of the prototype that was initially
named ruby-parser. Instead, this renames it to libprism to be
consistent with the actual name.

8600b06811
2023-11-14 16:22:03 -05:00
Kevin Newton
47163f9cf9 [ruby/prism] Rename suppress warnings to verbose
fbb30216ca
2023-11-03 14:54:13 +00:00
Kevin Newton
8587d9a8bf
[ruby/prism] Wire up options through the Java parser
13fa262669
2023-11-03 10:13:50 -04:00
Kevin Newton
6496591194
[ruby/prism] Rename serialization APIs for consistency
5a2252e3ac
2023-11-03 10:13:49 -04:00
Kevin Newton
05f5c545d2
[ruby/prism] Wire up options through the FFI API
f0aa8ad93b
2023-11-03 10:13:49 -04:00
Kevin Newton
79034fbd50 [ruby/prism] More Ruby docs
ca9a660f52
2023-11-01 13:10:29 -04:00
Kevin Newton
7bf3d9343f [ruby/prism] parse_inline_comments -> parse_comments
bd4d248fd6
2023-10-30 15:53:37 +00:00
Kevin Newton
c201dbc0ad [ruby/prism] Prism.parse_inline_comments
5b72f84480
2023-10-27 18:09:14 +00:00
Kevin Newton
5523a23469 [ruby/prism] Attach magic comments to the parse result
c7ef25a79a
2023-10-16 15:40:19 -07:00
Mau Magnaguagno
55a0d2c63b [ruby/prism] Avoid unnecessary delete_prefix in LibRubyParser.resolve_type
Only remove const prefix from non-pointer types.

97c9ffeb42
2023-10-16 10:39:44 +00:00