rapidyaml  0.16.0
parseandemitYAML,anddoitfast
Namespaces | Classes | Typedefs | Functions
Ints Event Handler
Parse utilities » Event Handlers

An event handler used by ParseEngine to creates an integer buffer with a very compact representation of the YAML tree in a source buffer. More...

Namespaces

 c4::yml::extra::ievt
 

Classes

struct  c4::yml::extra::EventHandlerInts
 A parser event handler that creates a compact representation of the YAML tree in a contiguous buffer of integers. More...
 

Typedefs

using c4::yml::extra::evt_size = int32_t
 data type for integer events size. More...
 

Functions

evt_size c4::yml::extra::estimate_events_ints_size (csubstr src)
 Read YAML source and, without undergoing a full parse, estimate the size of the integer buffer required for EventHandlerInts. More...
 
size_t c4::yml::extra::events_ints_to_testsuite (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz, substr evts_testsuite)
 Create a testsuite event string from integer events. More...
 
template<class Container >
void c4::yml::extra::events_ints_to_testsuite (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz, Container *evts_testsuite)
 Create a testsuite event string from integer events, writing into an output container. More...
 
template<class Container >
Container c4::yml::extra::events_ints_to_testsuite (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz)
 Create a testsuite event string from integer events, returning a new container with the result. More...
 
void c4::yml::extra::events_ints_print (csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz)
 Print integer events to stdout. More...
 

Detailed Description

An event handler used by ParseEngine to creates an integer buffer with a very compact representation of the YAML tree in a source buffer.

This feature is an extra and is not part of the main rapidyaml library.

This is meant for use by other programming languages, and supports container keys (unlike the ryml tree). It parses faster than the ryml tree parser, because the resulting data structure is much simpler.

The resulting integer buffer is a linear array of integers containing encoded YAML events (as a mask of ievt::EventBits), which in some cases (eg scalars) are followed by an encoded string (encoded as an offset and length to the parsed source buffer).

For example, parsing [a, bb, ccc] results in the following event buffer (grouped to highlight the event sequence structure):

source : [a, bb, ccc]
has a string........
| offset "a"
| | length "a"
| | |
event0 event1 event2 [ event3 "a"......|..|
| | | | | |
(start) +--------+-------+------------------+---------------+--+-----> (continued)
arr[i] : BSTR BDOC VAL_|BSEQ|FLOW VAL_|SCLR|PLAI..1..1
i : 0 1 2 3 4 5
has a string............. has a string.............
| offset "bb" | offset "ccc"
| | length "bb" | | length "ccc"
| | | | | |
event4 "bb"..........|..| event5 "ccc".........|..|
| | | | | |
(cont)--> -----+--------------------+--+--------------+--------------------+--+-----> (continued)
arr[i] : VAL_|SCLR|PLAI|PSTR..4..2 VAL_|SCLR|PLAI|PSTR..8..3
i : 6 | 7 8 9 | 10 11
| |
prev event has string prev event has string
(to get to prev, jump (to get to prev, jump
back 3 slots: ie 6->3) back 3 slots: ie 9->6)
event6 ] event7 event8
| | |
(cont)--> -----+-------------+--------+-----| (end)
arr[i] : ESEQ|PSTR EDOC ESTR
i : 12 | 13 14
|
prev event has string
(to get to it, jump
back 3 slots: ie 12->9)

Note that the buffer contains both events and strings encoded as integer pairs. That is, events that have an associated string are immediately followed by two integers providing the offset and length of that string in the source buffer. (In the example above, this happens in the events for the strings a, bb, and ccc at positions 3, 6 and 9, respectively).

The flag ievt::PSTR and the mask ievt::WSTR are provided to enable easier iteration over the array: you can use them to test for presence of a string when iterating over the array.

The flag ievt::PSTR announces that an event is preceded by a string. That is, the previous event has a string, so that when this flag is found while iterating right-to-left, a jump of -3 should be used to get at the bitmask of the previous event. (In the example above, this flag is present for the events for bb and ccc, but not a because it is not preceded by a string).

Likewise, to signify that the current event is followed by a string, there is the mask ievt::WSTR, which is a mask of all the flags of events that have a string: ievt::SCLR, ievt::ALIA, ievt::ANCH and ievt::TAG_. While iterating left-to-right in the array, presence of any of the bits in the mask ievt::WSTR means that a jump of +3 should be employed to get at the bitmask of the next event.

Here's another example with the result of parsing a: bb

Typedef Documentation

using c4::yml::extra::evt_size = typedef int32_t

data type for integer events size.

This is set to an int32_t integer to allow compatibility with a wide range of processing languages.

Definition at line 176 of file event_handler_ints.hpp.

Function Documentation

evt_size c4::yml::extra::estimate_events_ints_size ( csubstr  src)

Read YAML source and, without undergoing a full parse, estimate the size of the integer buffer required for EventHandlerInts.

This estimation is meant to exceed the actual number of required events.

Note
This function must overpredict. It does so for every case in the hundreds/thousands of extensive tests of rapidyaml – both for the YAML test suite and the internal cases. If you find a case where that does not hold, it is a bug. Please report it at https://github.com/biojppm/rapidyaml/issues!

Definition at line 25 of file event_handler_ints.cpp.

Referenced by main().

26 {
27  evt_size count = 7; // BSTR + BDOC + =VAL + EDOC + ESTR
28  for(size_t i = 0; i < src.len; ++i)
29  {
30  switch(src.str[i])
31  {
32  case ':': // this has strings preceding/following it
33  case ',': // overestimate, assume map
34  case '%': // assume TAGD->string + TAGV->string
35  count += 6;
36  break;
37  // these have (or are likely to have) a string following it
38  case '-':
39  case '&':
40  case '*':
41  case '<':
42  case '!':
43  case '\'':
44  case '"':
45  case '|':
46  case '>':
47  case '\n':
48  count += 3;
49  break;
50  case '[':
51  case ']':
52  count += 4;
53  break;
54  case '{':
55  case '}':
56  count += 7;
57  break;
58  case '?':
59  count += 5;
60  break;
61  }
62  }
63  return count;
64 }
c4::yml::extra::evt_size
int32_t evt_size
data type for integer events size.
Definition: event_handler_ints.hpp:176
size_t c4::yml::extra::events_ints_to_testsuite ( csubstr  parsed_yaml,
csubstr  arena,
ievt::evt_bits const *  evts_ints,
ievt::evt_bits  evts_ints_sz,
substr  evts_testsuite 
)

Create a testsuite event string from integer events.

This overload receives a buffer where the string is to be written, and returns the size needed to accomodate the result. The size of the buffer is strictly respected. The caller must check that the returned size is smaller than the buffer's size to ensure that the result is complete. If that's not the case, the user must resize the buffer and call again.

Definition at line 33 of file ints_to_testsuite.cpp.

Referenced by c4::yml::extra::events_ints_to_testsuite().

38 {
39  auto getstr = [&](ievt::evt_bits i){
40  bool in_arena = evts_ints[i] & ievt::AREN;
41  csubstr region = !in_arena ? parsed_yaml : arena;
42  return region.sub((size_t)evts_ints[i+1], (size_t)evts_ints[i+2]);
43  };
44  size_t sz = 0;
45  auto append = [&](csubstr s){
46  size_t next = sz + s.len;
47  if (s.len && (next <= evts_test_suite.len && evts_test_suite.len))
48  memcpy(evts_test_suite.str + sz, s.str, s.len);
49  sz = next;
50  };
51  bool has_tag = false;
52  csubstr tag;
53  auto maybe_append_tag = [&]{
54  if(has_tag)
55  {
56  if(tag.begins_with('<'))
57  {
58  append(" ");
59  append(tag);
60  }
61  else
62  {
63  RYML_ASSERT_BASIC_(tag.begins_with('!'));
64  append(" <");
65  append(tag);
66  append(">");
67  }
68  }
69  has_tag = false;
70  };
71  bool has_anchor = false;
72  csubstr anchor;
73  auto maybe_append_anchor = [&]{
74  if(has_anchor)
75  {
76  append(" &");
77  append(anchor);
78  }
79  has_anchor = false;
80  };
81  auto append_cont = [&](csubstr evt, csubstr style){
82  append(evt);
83  if(style.len)
84  {
85  append(" ");
86  append(style);
87  }
88  maybe_append_anchor();
89  maybe_append_tag();
90  append("\n");
91  };
92  auto append_esc = [&](csubstr str){
93  substr buf = sz <= evts_test_suite.len ? evts_test_suite.sub(sz) : evts_test_suite.last(0);
94  sz += escape_scalar(buf, str);
95  append("\n");
96  };
97  auto append_val = [&](csubstr evt, csubstr val){
98  append("=VAL");
99  maybe_append_anchor();
100  maybe_append_tag();
101  append(" ");
102  append(evt);
103  append_esc(val);
104  };
105  ievt::evt_bits evt = 0;
106  for(ievt::evt_bits i = 0; i < evts_ints_sz; i += (evt & ievt::WSTR) ? 3 : 1)
107  {
108  evt = evts_ints[i];
109  if(evt & ievt::SCLR)
110  {
111  csubstr s = getstr(i);
112  if(evt & ievt::SQUO)
113  append_val("'", s);
114  else if(evt & ievt::DQUO)
115  append_val("\"", s);
116  else if(evt & ievt::LITL)
117  append_val("|", s);
118  else if(evt & ievt::FOLD)
119  append_val(">", s);
120  else //if(evt & ievt::PLAI)
121  append_val(":", s);
122  }
123  else if((evt & ievt::BSEQ) == ievt::BSEQ)
124  {
125  if(evt & ievt::FLOW)
126  append_cont("+SEQ", "[]");
127  else
128  append_cont("+SEQ", "");
129  }
130  else if((evt & ievt::ESEQ) == ievt::ESEQ)
131  {
132  append("-SEQ\n");
133  }
134  else if((evt & ievt::BMAP) == ievt::BMAP)
135  {
136  if(evt & ievt::FLOW)
137  append_cont("+MAP", "{}");
138  else
139  append_cont("+MAP", "");
140  }
141  else if((evt & ievt::EMAP) == ievt::EMAP)
142  {
143  append("-MAP\n");
144  }
145  else if(evt & ievt::ALIA)
146  {
147  append("=ALI *");
148  append(getstr(i));
149  append("\n");
150  }
151  else if(evt & ievt::TAG_)
152  {
153  has_tag = true;
154  tag = getstr(i);
155  }
156  else if(evt & ievt::ANCH)
157  {
158  has_anchor = true;
159  anchor = getstr(i);
160  }
161  else if((evt & ievt::BDOC) == ievt::BDOC)
162  {
163  if(evt & ievt::EXPL)
164  append("+DOC ---\n");
165  else
166  append("+DOC\n");
167  }
168  else if((evt & ievt::EDOC) == ievt::EDOC)
169  {
170  if(evt & ievt::EXPL)
171  append("-DOC ...\n");
172  else
173  append("-DOC\n");
174  }
175  else if((evt & ievt::BSTR) == ievt::BSTR)
176  {
177  append("+STR\n");
178  }
179  else if((evt & ievt::ESTR) == ievt::ESTR)
180  {
181  append("-STR\n");
182  }
183  }
184  return sz;
185 }
c4::yml::extra::ievt::BDOC
begin doc (+DOC in test suite events)
Definition: event_handler_ints.hpp:210
c4::yml::extra::ievt::BSEQ
begin seq (+SEQ in test suite events)
Definition: event_handler_ints.hpp:204
c4::yml::extra::ievt::WSTR
WithSTRing: mask of all events that encode a string following the event. For such events...
Definition: event_handler_ints.hpp:251
c4::yml::extra::ievt::FOLD
scalar: block folded (>)
Definition: event_handler_ints.hpp:228
str
const char * str
Definition: node_type.cpp:70
c4::yml::escape_scalar
size_t escape_scalar(substr buffer, csubstr scalar, bool keep_newlines=false)
Escape a scalar to an existing buffer, using escape_scalar_fn.
Definition: escape_scalar.hpp:241
c4::yml::extra::ievt::LITL
scalar: block literal (|)
Definition: event_handler_ints.hpp:227
c4::basic_substring::sub
basic_substring sub(size_t first) const noexcept
return [first,len[
Definition: substr.hpp:502
c4::yml::extra::ievt::SQUO
scalar: single-quoted (')
Definition: event_handler_ints.hpp:225
c4::substr
basic_substring< char > substr
a mutable string view
Definition: substr.hpp:2355
c4::csubstr
basic_substring< const char > csubstr
an immutable string view
Definition: substr.hpp:2356
c4::yml::extra::ievt::ESEQ
end seq (-SEQ in test suite events)
Definition: event_handler_ints.hpp:205
c4::yml::extra::ievt::ESTR
end stream (-STR in test suite events)
Definition: event_handler_ints.hpp:209
c4::basic_substring::len
size_t len
the length of the substring
Definition: substr.hpp:218
c4::yml::extra::ievt::EDOC
end doc (-DOC in test suite events)
Definition: event_handler_ints.hpp:211
c4::yml::extra::ievt::DQUO
scalar: double-quoted ("")
Definition: event_handler_ints.hpp:226
c4::yml::extra::ievt::BMAP
begin map (+MAP in test suite events)
Definition: event_handler_ints.hpp:206
c4::yml::extra::ievt::FLOW
container: flow: [] for seqs or {} for maps
Definition: event_handler_ints.hpp:229
c4::yml::extra::ievt::evt_bits
int32_t evt_bits
data type for integer events bits.
Definition: event_handler_ints.hpp:183
c4::yml::extra::ievt::AREN
Special flag to mark events whose string was placed in the arena. This happens when the filtered stri...
Definition: event_handler_ints.hpp:246
c4::yml::extra::ievt::EXPL
--- (with BDOC) or ... (with EDOC)
Definition: event_handler_ints.hpp:202
c4::yml::extra::ievt::SCLR
scalar (=VAL in test suite events)
Definition: event_handler_ints.hpp:214
c4::yml::extra::ievt::EMAP
end map (-MAP in test suite events)
Definition: event_handler_ints.hpp:207
c4::yml::extra::ievt::BSTR
begin stream (+STR in test suite events)
Definition: event_handler_ints.hpp:208
template<class Container >
void c4::yml::extra::events_ints_to_testsuite ( csubstr  parsed_yaml,
csubstr  arena,
ievt::evt_bits const *  evts_ints,
ievt::evt_bits  evts_ints_sz,
Container *  evts_testsuite 
)

Create a testsuite event string from integer events, writing into an output container.

Definition at line 39 of file ints_to_testsuite.hpp.

44 {
45  size_t len = events_ints_to_testsuite(parsed_yaml, arena, evts_ints, evts_ints_sz, to_substr(*evts_testsuite));
46  if(len > evts_testsuite->size())
47  {
48  evts_testsuite->resize(len);
49  len = events_ints_to_testsuite(parsed_yaml, arena, evts_ints, evts_ints_sz, to_substr(*evts_testsuite));
50  }
51  evts_testsuite->resize(len);
52 }
c4::to_substr
substr to_substr(char(&s)[N]) noexcept
Definition: substr.hpp:2376
c4::yml::extra::events_ints_to_testsuite
Container events_ints_to_testsuite(csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz)
Create a testsuite event string from integer events, returning a new container with the result...
Definition: ints_to_testsuite.hpp:58
template<class Container >
Container c4::yml::extra::events_ints_to_testsuite ( csubstr  parsed_yaml,
csubstr  arena,
ievt::evt_bits const *  evts_ints,
ievt::evt_bits  evts_ints_sz 
)

Create a testsuite event string from integer events, returning a new container with the result.

Definition at line 58 of file ints_to_testsuite.hpp.

62 {
63  Container ret;
64  events_ints_to_testsuite(parsed_yaml, arena, evts_ints, evts_ints_sz, &ret);
65  return ret;
66 } // LCOV_EXCL_LINE
c4::yml::extra::events_ints_to_testsuite
Container events_ints_to_testsuite(csubstr parsed_yaml, csubstr arena, ievt::evt_bits const *evts_ints, ievt::evt_bits evts_ints_sz)
Create a testsuite event string from integer events, returning a new container with the result...
Definition: ints_to_testsuite.hpp:58
void c4::yml::extra::events_ints_print ( csubstr  parsed_yaml,
csubstr  arena,
ievt::evt_bits const *  evts,
ievt::evt_bits  evts_sz 
)

Print integer events to stdout.

Definition at line 103 of file ints_utils.cpp.

Referenced by main().

104 {
105  char buf[200];
106  for(ievt::evt_bits evtpos = 0, evtnumber = 0;
107  evtpos < evts_sz;
108  ++evtnumber,
109  evtpos += ((evts[evtpos] & ievt::WSTR) ? 3 : 1))
110  {
111  ievt::evt_bits evt = evts[evtpos];
112  csubstr flags = ievt::to_str_sub(buf, evt);
113  printf("[%d][%d] %.*s(0x%x)", evtnumber, evtpos, (int)flags.len, flags.str, evt);
114  if (evt & ievt::WSTR)
115  {
116  bool in_arena = evt & ievt::AREN;
117  csubstr region = !in_arena ? parsed_yaml : arena;
118  bool safe = (evts[evtpos + 1] >= 0)
119  && (evts[evtpos + 2] >= 0)
120  && (evts[evtpos + 1] <= (ievt::evt_bits)region.len) // NOLINT
121  && (evts[evtpos + 2] <= ((ievt::evt_bits)region.len - evts[evtpos + 1]));
122  const char *str = safe ? (region.str + evts[evtpos + 1]) : "ERR!!!";
123  ievt::evt_bits len = safe ? evts[evtpos + 2] : 6;
124  printf(": %d [%d]~~~%.*s~~~", evts[evtpos+1], evts[evtpos+2], len, str);
125  if(in_arena)
126  printf(" (arenasz=%zu)", arena.len);
127  else
128  printf(" (srcsz=%zu)", parsed_yaml.len);
129  }
130  printf("\n");
131  }
132 }
c4::yml::extra::ievt::WSTR
WithSTRing: mask of all events that encode a string following the event. For such events...
Definition: event_handler_ints.hpp:251
flags
EventBits flags
Definition: ints_utils.cpp:27
c4::yml::extra::ievt::to_str_sub
csubstr to_str_sub(substr buf, ievt::evt_bits flags)
Convert bit mask of ievt::EventBits to text.
Definition: ints_utils.cpp:80
c4::csubstr
basic_substring< const char > csubstr
an immutable string view
Definition: substr.hpp:2356
c4::yml::extra::ievt::evt_bits
int32_t evt_bits
data type for integer events bits.
Definition: event_handler_ints.hpp:183
c4::yml::extra::ievt::AREN
Special flag to mark events whose string was placed in the arena. This happens when the filtered stri...
Definition: event_handler_ints.hpp:246
str
const char * str
Definition: ints_utils.cpp:27