|
NVRAM Module
|
|
| Phase
1 Rewrite |
1 Overview
In the beginning nvram was intended to be used for router configuration
info only, and the data was allowed to grow until all the memory was
used up. Over time other uses for nvram were found and implemented
without a full review of the current design. This has lead to an nvram
implementation that is full of hacks and patches that the current
design cannot handle reliably, yet the number of requirements and
feature requests continue to grow. To meet these demands the typical
response has been to increase the amount of nvram memory. This approach
simply does not scale and the current implementation has reach it's
limit.
This specification proposes to re-implement nvram as a well defined
module with managed dependencies and a clean and rigidly enforced api.
The design will be robust enough to accommodate the current
requirements even as the amount of memory used continues to increase,
and will be versatile enough to accommodate new uses as they arise.
More precisely, the new implementation will simulate a real file system
with individual files and protected partitions. It will handle the
special needs of nvram internally, giving the developer the tools to
implement any number of algorithms to meet various design requirements
without the need to know any details of the nvram implementation.
The longer range goal is to componentize the nvram module. This will
put the implementation out of reach and present just the api to the
rest of IOS. Only then can the nvram module be protected from feature
creep and code evolution. Thus nvram would be expected to stabilize and
become more robust over time.
1.1 Problem Analysis
The current implementation of nvram allocates a contiguous block of
memory, and it is up to the developer to parse the memory into it's
constituent parts. For historical reasons the main part of nvram is the
startup configuration. At some point this "public" configuration data
was extended to include some "private" configurations, but the memory
space was not divided into sections to accommodate this new type of
data. Instead the private configs were simply appended to the end of
the startup configs, meaning that the calling process must keep track
of the configuration boundaries.
More recently a third type of data was added, called persistent data,
which was not specific to router configuration, but which needed to be
stored securely and locally on the router. Again, no memory space was
craved out for this new type of data but instead it was simply added to
the very end of nvram memory.
One of the characteristics of nvram is that it's actually a serial
storage
device mapped as an ifs file. The "ram" part of nvram is a misnomer,
and serial access means that all of the configuration data must be read
into a buffer where it can be modified and then written out as a
contiguous block, overwriting all of the previous data. Thus, adding
data to the startup config requires shifting some part of the buffered
data to the right in
order to make room, and deleting means shifting the buffered data to
the left in order to
overwrite the old data. All of this is done in the system memory buffer
and not
in nvram.
Another characteristic of nvram is that it is non-removable. The nvram
memory hardware is part of the router, and that fact has certain
security implications. Thus nvram has become the preferred place to
store "persistent" data that remains securely with the router at all
times and across router reloads.
In order to avoid complicating the process for storing configuration
data, the
persistent data is stored in blocks starting at the end of nvram memory
space and working backwards. This obviously does not scale, and the
inevitable question becomes which is more "persistent" when nvram is
full ?? The fact is that the nvram memory space is not controlled and
the configuration data is not prevented from using all of nvram memory,
and it will simply overwrite the persistent data if the memory is
needed.
Thus the current implementation of nvram is a programming
challenge because it leads to
conflicts when memory is at a premium, and the calling process must
detect a conflict in order to issue a message if the persistent data is
about to be overwritten. It is also up to the calling process to deal
with the fact that the
configuration data is actually two sets of data, public and private,
and must keep track of that boundary when making modifications.
Another complication is that because nvram is a serial device a read or
write operation must be allowed to suspend in order to prevent
potentially large
data transfers form hogging the cpu. During that suspend the nvram ifs
file is
held open, and if another process tries to access any of the config or
persistent data it fails with a "busy" error. It's then up to the
calling process to either suspend and try again later
or just give up.
Finally, the checksum calculation has created a performance problem.
The checksum is primarily used to validate nvram and verify that it has
not been corrupted between writes and between router reloads. In order
to accomplish this,
all of nvram must be checksum'd after every write operation because
there are no stable boundaries within it's memory space and so no way
to tell if only a portion needs to be verified. On
platforms with large amounts of nvram this can mean that saving even a
simple configuration change can take an unacceptable amount of time to
complete.
1.2 Modular Solution
The nvram module partitions its memory space into file-blocks and
protects those file's boundaries. A file is then accessed using nvram
api
functions that open, read, write, seek, configure and close the file.
What a calling process gets is a buffer of data of a specified size
from the
specified nvram file. The buffer can be searched, modified, extended,
deleted, etc, and then written back to nvram.
There are three major differences between this and the old
implementation:
1. calling process does not have
to
consider the boundary between startup configs and private configs.
2. calling process does not have to deal with conflicts between
config and persistent data.
3. calling process can request any portion of the file data
instead of
the whole thing.
For example, a process could request all of the persistent data, or set
up a loop to grab some buffer size one at a time, then search, modify,
extend and delete various sections of the buffer, and then write only
the buffers that were modified back to the
persistent file. The nvram algorithms will figure out what data blocks
need
to be inserted, what blocks need to be deleted and what blocks to be
modified.
In any case, the data will not be written outside of the file's
boundaries.
These changes are also useful for accessing the startup config file,
which is uploaded into system memory at runtime, and is passed back to
the nvram code as a contiguous buffer at "savetime". The nvram module
will parse the buffered data into blocks, and only those blocks that
have changed will be written back to nvram and have it's checksum
re-generated.
The new nvram module does not hold the nvram ifs file open between
operations. Calling the "open" api function creates and initializes a
file access node but does not actually open the nvram ifs file. A
"read" or "write" function call opens, reads and closes the nvram ifs
file before returning. This allows the calling process to suspend for
any amount of time, for example waiting on "auto-more", and yet not
block other processes from accessing nvram. Although a large data
transfer may still generate a suspend within the nvram module, it's
guaranteed an nvram suspend
will to be scheduled to run again as soon as possible.
1.3 New Characteristics
The new nvram module has the following new characteristics:
- Assumes that a write starts where the last read started, unless
there was a seek. Thus a read-modify-write operation will not require a
seek to reposition the file pointer.
- Doesn't start writing until the first modification is detected.
If the length changes, in "legacy" mode the rest of the file is shifted
accordingly. When in "file" mode (i.e. not supporting the old memory
map) the module can insert non-contiguous blocks of nvram memory where
needed. In this way the rest of the file data doesn't have to be
shifted
to make room.
- Generates a checksum for each individual memory block instead of
for all of nvram.
The following are some examples on using the api functions. Note that
section 4.3 lists the set of possible error codes returned by the nvram
module, and section 4.4 gives a detailed description of the api
functions and the error codes each can return.
1.3 Compatibility Issues
It is not realistic to expect to re-implement all of nvram across all
platforms and then require all the processes that use nvram to update
their code. Nor is it realistic to expect all router customers to
update their images for a feature enhancement that may not be useful to
some. It's
more likely that over time developers and customers will use the new
nvram api when the current limitations become an obstacle.
This means that the new nvram module must exist alongside the old
implementation, although this is problematic because the new nvram
module
changes the way data is stored. In theory the new code could just map
the files to match the old memory space, except that the old private
config file boundary changes with the size of the startup config. In
otherwords, the location of the private config file keeps moving.
The nvram module must also provide the tools to migrate to the new
paradigm when all of IOS has made the conversion. This would be version
specific and would necessitate the option to convert back to the old
memory layout before an older IOS version is loaded.
This approach would require little if any change to the old code base,
and puts the burden on the new implementation.