Parsing automatically??

[lilzz]

Code: Select all

 std::string device;
  int width;
  int height;
  int n_tiles;
  int n_nets;
  int n_global_nets;
  std::map<int, int> net_global;
  std::map<std::string, Package> packages;
  std::map<Location, int> loc_pin_glb_num;

ibstream &ibs
  ibs >> device
      >> width
      >> height
      >> n_nets
      >> packages
      >> loc_pin_glb_num

There's database file, and the code use ibstream to read in the database file, but how does it know which data should going to which variables?? From above , it looks like all the data has been parsed automatically by ibs and all the data have been sorted out and go to corresponding variables. Am I missing something?

[Paeryn]

Without seeing the code for ibstream we can't say if/how it parses the data. Whatever is using it seems to know that the data is coming out in that order so either whoever wrote it knows that is the order or something told ibstream the order to present the data (either the program calling it or it's embedded in the database it is reading or somesuch).

[lilzz]

nothing special about the ibstream . I don't see how it knows how to parse the data.

Code: Select all

class ibstream
{
private:
  std::istream &is;
  
public:
  ibstream(std::istream &is_)
    : is(is_)
  {}
  
  void read(char *p, size_t n)
  {
    is.read(p, n);
    size_t rn = is.gcount();
    if (is.bad()
	|| rn != n)
      fatal(fmt("std::istream::read: " << strerror(errno)));
  }
};
 
inline ibstream &operator>>(ibstream &ibs, bool &x)
{
  ibs.read(reinterpret_cast<char *>(&x), sizeof(x));
  // *logs << "bread bool " << x << "\n";
  return ibs;
}

inline ibstream &operator>>(ibstream &ibs, char &x)
{
  ibs.read(reinterpret_cast<char *>(&x), sizeof(x));
  // *logs << "bread char " << x << "\n";
  return ibs;
}

inline ibstream &operator>>(ibstream &ibs, unsigned char &x)
{
  ibs.read(reinterpret_cast<char *>(&x), sizeof(x));
  // *logs << "bread unsigned char " << x << "\n";
  return ibs;
}

template<typename T> ibstream &
bread_signed_integral_type(ibstream &ibs, T &x)
{
  x = 0;
  constexpr int T_bits = CHAR_BIT*sizeof(T);
  int shift = 0;
  for (;;)
    {
      char b;
      ibs >> b;
      x |= (int)(b & 0x7f) << shift;
      shift += 7;
      if (! (b & 0x80))
	{
	  if (shift < T_bits
	      && (b & 0x40))
	    x = (x << (T_bits - shift)) >> (T_bits - shift);
	  break;
	}
    }
  return ibs;
}

template<typename T> ibstream &
bread_unsigned_integral_type(ibstream &ibs, T &x)
{
  x = 0;
  unsigned shift = 0;
  for (;;)
    {
      char b;
      ibs >> b;
      x |= ((unsigned)(b & 0x7f) << shift);
      shift += 7;
      if (! (b & 0x80))
	break;
    }
  return ibs;
}

inline ibstream &operator>>(ibstream &ibs, short &x)
{
  return bread_signed_integral_type<short>(ibs, x);
}

inline ibstream &operator>>(ibstream &ibs, unsigned short &x)
{
  return bread_unsigned_integral_type<unsigned short>(ibs, x);
}

inline ibstream &operator>>(ibstream &ibs, int &x)
{
  return bread_signed_integral_type<int>(ibs, x);
}

inline ibstream &operator>>(ibstream &ibs, unsigned &x)
{
  return bread_unsigned_integral_type<unsigned>(ibs, x);
}

inline ibstream &operator>>(ibstream &ibs, long &x)
{
  return bread_signed_integral_type<long>(ibs, x);
}

inline ibstream &operator>>(ibstream &ibs, unsigned long &x)
{
  return bread_unsigned_integral_type<unsigned long>(ibs, x);
}

inline ibstream &operator>>(ibstream &ibs, std::string &s)
{
  size_t n;
  ibs >> n;
  s.resize(n);
  ibs.read(&s[0], n);
  s[n] = 0;
  return ibs;
}

template<class T> ibstream &
operator>>(ibstream &ibs, std::vector<T> &v)
{
  size_t n;
  ibs >> n;
  v.resize(n);
  for (size_t i = 0; i < n; ++i)
    ibs >> v[i];
  return ibs;
}

template<class T> ibstream &
operator<<(ibstream &ibs, std::set<T> &s)
{
  s.clear();
  size_t n;
  ibs >> n;
  for (size_t i = 0; i < n; ++i)
    {
      T x;
      ibs >> x;
      s.insert(x);
    }
  return ibs;
}

template<typename K, typename V, typename C> ibstream &
operator>>(ibstream &ibs, std::map<K, V, C> &m)
{
  m.clear();
  size_t n;
  ibs >> n;
  for (size_t i = 0; i < n; ++i)
    {
      std::pair<K, V> x;
      ibs >> x;
      m.insert(x);
    }
  return ibs;
}


template<typename F, typename S> inline ibstream &
operator>>(ibstream &ibs, std::pair<F, S> &p)
{
  return ibs >> p.first >> p.second;
}

template<typename F, typename S, typename T> inline ibstream &
operator>>(ibstream &ibs, std::tuple<F, S, T> &t)
{
  return ibs >> std::get<0>(t)
	     >> std::get<1>(t)
	     >> std::get<2>(t);
}

[Paeryn]

From a quick scan of the code it's a basic bytestream, it just reads the file/stream into the variables, so if reading to char it reads one byte, to an int it reads four bytes etc. You have to know what order and size it was written and just read the data in the same order and with the same size variables. If you try reading a char where an int was written then everything from then on will be wrong.

The parsing is in the code you first posted. The first item is a string and goes to device, second is an int that goes to width etc.

[AndyD]

@lilzz, Serialization and Unserialization from isocpp.org may be of interest to you.

[rurwin]

The code is all there. You've posted it all.

It reads each character one by one and interprets them according to whatever type it's trying to read.

If the type is a char it just reads a single character into the variable.
If the type is a bool it reads sizeof(bool) characters into the variable. (That's a bug if this is supposed to be workable across platforms which, considering the trouble they go to for ints, seems likely.)
If it's an int it reads bytes and puts them into the int until it gets to one with the top bit set. So each character in the file provides 7 bits. There's some fiddling I can't follow without paper and pencil for signed ints.
If it's a string then it reads a length number first then that number of bytes are read into the string.
Vectors, sets and maps work the same except the elements are read as the types they are and each map element is read as a pair.
pairs and tuples are read as just their individual elements of whatever types they are.

So the file doesn't hold text; it holds binary data that is only understandable if you know what values of what types occur in what order.

Code: Select all

    ibstream &ibs
      ibs >> device
          >> width
          >> height
          >> n_nets
          >> packages
          >> loc_pin_glb_num

The first read is device. device is a string. So it reads a string into device. The next is width, which is an int. So it reads an int into width. And so on.

[lilzz]

The code actually like this

Code: Select all

ibs >> device
      >> width
      >> height
      >> n_nets
      >> packages
      >> loc_pin_glb_num
      >> iolatch
      >> ieren
      >> extra_bits
      >> gbufin
      >> tile_colbuf_tile
      >> tile_type
      >> net_names
      >> tile_nets_idx 
      >> tile_nonrouting_cbits
      >> extra_cell_tile
      >> extra_cell_type
      >> extra_cell_mfvs
      >> n_cells
      >> cell_type
      >> cell_location
      >> cell_type_cells
      >> switches
      >> tile_cbits_block_size;

The database is like this

Code: Select all

.device 1k 14 18 26562

.pins tq144
1 0 14 1
10 0 11 0
101 13 13 0
102 13 13 1
104 13 14 0
105 13 14 1
106 13 15 0
107 13 15 1
11 0 10 1
112 12 17 1
113 12 17 0
114 11 17 1
115 11 17 0
116 10 17 1
117 10 17 0
118 9 17 1
119 9 17 0
12 0 10 0
120 8 17 1
121 8 17 0
122 7 17 1
128 7 17 0
129 6 17 1
134 5 17 1
135 5 17 0
136 4 17 1
137 4 17 0
138 3 17 1
139 3 17 0
141 2 17 1
142 2 17 0
143 1 17 1
144 1 17 0
19 0 9 1


.gbufin
0 8 6
0 9 3
6 0 5
6 17 4
7 0 0
7 17 1
13 8 7
13 9 2

.gbufpin
13 8 1 0
0 8 1 1
7 17 0 2
7 0 0 3
0 9 0 4
13 9 0 5
6 0 1 6
6 17 1 7

.iolatch
0 7
5 0
8 17
13 10

.ieren
0 2 0 0 2 1
0 2 1 0 2 0
0 3 0 0 3 1
0 3 1 0 3 0
0 4 0 0 4 1
0 4 1 0 4 0
0 5 0 0 5 1
0 5 1 0 5 0
0 6 0 0 6 1
0 6 1 0 6 0
0 8 0 0 8 1
0 8 1 0 8 0
0 9 0 0 9 1
0 9 1 0 9 0
0 10 0 0 10 1
0 10 1 0 10 0
0 11 0 0 11 1
0 11 1 0 11 0
0 12 0 0 12 1
0 12 1 0 12 0
0 13 0 0 13 1
0 13 1 0 13 0
0 14 0 0 14 1
0 14 1 0 14 0
1 0 0 1 0 0
1 0 1 1 0 1
1 17 0 1 17 0
1 17 1 1 17 1



.colbuf
0 4 0 0
0 4 0 1
0 4 0 2
0 4 0 3
0 4 0 4
0 5 0 5
0 5 0 6
0 5 0 7
0 5 0 8
0 12 0 9
0 12 0 10
0 12 0 11
0 12 0 12
0 13 0 13
0 13 0 14
0 13 0 15
0 13 0 16

If trying to match by position it won't match at all. It need to identify the section such .device, .pin, .iolatch...etc. Inside each section there is long list of data, if you try to match by position, then it would Fail! The identifier iolatch is before the gbufin but in databse iolatch is after gbufin, It needs to correctly identify the right section. I don't see ibstream able to do that. This is just one example, there are many sections the order is different .

[Paeryn]

The main program more than likely knows what section it is supposed to be reading next, at that point you gave it knows that the next entry will be a device section and so attempts to read a device section. ibstream itself knows nothing of what it is reading.

[rurwin]

You are mixing up two different formats.

I assume this is arachne-pnr, at least that's where the code you've shown appears to come from.
For those who want to follow along:https://github.com/cseed/arachne-pnr/bl ... /chipdb.cc

The textual output that you listed just above is produced by ChipDB::dump(std::ostream &s) and read by ChipDBParser::parse(), where you will find the parsing functions you are looking for in all their many lines of code.

However the scrap of code you listed in the original post is from ChipDB::bread(ibstream &ibs) which reads a binary file format that is written by ChipDB::bwrite(obstream &obs).