Record, record, and record once again.

Record, record and record once again.

Files attached to the article:

file1

file2

Let's talk about the records. About those problems and nuances with which we are regularly facing, about those possibilities which SA posesses in the last versions.

Own SA loader has appeared by some reasons. One of the reasons - is widely spread I/Q format, and with other, strangely enough, rather poor quality of records in this format became one of the reasons of its working out, on the one hand.

Poor quality of records in general is a global problem, we considered it here earlier, so there is no sense to repeat, I wish to mention some questions which are not so obvious.

Several I/Q records from recourse http://www.iw3aut.altervista.org/files/samples/index.php

At loading of these files in SA (look attached files), you will be surprised by extremely low level of the records.

Althought, the records apparently, should show all power and possibilities of SDR technology. Unfortunately the result is absolutely opposite. It is impossible to consider neither samples, nor examples of qualitative work.

Level of the record is so low, that at attempts to increase it, there is rather rough granularity and the resulting signal has worse quality than 8 bit record. I would like to remind that 8 bit records are considered by default as unsuitable for tasks of the signal analysis. Such recordS are more suitable for speech digitalization in narrow enought bar.

Dynamic range of such "signals" is extremely low. Spectrally such records look very plainly and lifelessly, I name them as "dead" records, the background noise level, which caused by rough digitization on level, is very high. Such records represent the extremely doubtful material for tasks of signals identification and analysis, in spite of the fact that it I/Q format should provide the highest quality.

This problem is very characteristic for SDR receivers, we have faced it about 4 years ago, however attempts to pay attention to the very bad quality of material for the analysis and identification, hadn't found special understanding, at least the stream of an illiquid material has not decreased at all.

It is curious enough to observe the bad quality of records today, while quality of SDR receivers is constantly grows, digit capacity ADC raises, and the problem is still exists and not going to disappear..

It is may to seem that the heart the problem is insufficient amplification in paths of processing of signal of SDR receivers. In DSP all is co-ordinated in a single whole unit, nothing is taken from anywhere, and nothing disappears into anywhere. If the main ADC has digit capacity in 16 bits then these 16 bits will be allocated on whole length of digitization bar, and if this bar, for example is 40 Mhz, then it is necessary ~600-620 Hz for one position. It means, that if after such ADC you try to select bar in 10 Khz from 40 Mhz, then there is never be high-grade 16 bits, and will be only approximately 20-40 (position)/levels.

It is rough enough example but it reflexts reality. Althought it is only of one possible problems which is linked with digit capacity ADC, which are used in modern SDR receivers.

This problem can be masked easily enought by various filters, switching to higher digit capacity or to arithmetics with a floating point after ADC, but it does not change a real state of affairs, and forces developers to solve uneasy questions.

There are lot of powerfull SDR recievers with powerfull ADC represented today, they are providing a signal's output in 24 or 32 bit format. It would Seem in such situation, that quality of records should be good by default, however it is far not always so.

Dynamic range of 16 bit records is approximately 96 Db, 24 bit is nearby 145 db, 32 bit is about 193 db. If to consider real-life formats which are widely used for records,then for 32 bit numbers floating-point dynamic range will be 700 db, and for 64 bit 7000-8000 db. These are fantastic values of a dynamic range, to display which in a reality on a spectrum is almost not possible, you simply will not see a difference between 650 and 680 db, otherwise the spectrum will remind small cutted chaotic colour salad, which will be hardly possible to understand. qualitative mapping of a complete dynamic range is a certain problem already on 16 bits, and to play back 140-145 db for 24 bit records, very and very uneasy.

Smoothly we approach to the problem. The big dynamic range though guarantees good quality, but does not guarantee automatic solution of all problems, an ability to use all potential power of equipment remains for the person, however not all understand that, and worst of all, most of people do not wish to understand these things that is a little surprising in general.

Typical, even classical example. There is the record 24 bit, that should provide a dynamic range in 140-145 dB.

It is well visible that random impulse interference, completely brings to nothing all advantages of 24 bit formats, as in this record on the main signal we have only about 5000-7000 (!) level/positions, from approximately 16700000 possible. The most part of the dynamic range is spent on exellent representation of the random, single powerful impulse., to the prejudice of other data. And considering that in this record the common bar is 96 Khz and the signal demanding the analysis occupies 3 KHz, then the chances to receive a picture which we got earlier are very great.

This problem is getting worse, because in various cases for the various reasons, 16 bit records are especcialy demanded, and in this case direct conversion makes a bad turn. Usually sound editors are used for these purposes. The file is loaded as 24 or 32 bit and saved as 16. At such approach, the record before saving is scaled, and any program fairly tries to spend this scaling with efforts to provide maximum identity with the original.

In this case, even if the original provides sufficient quality for the analysis, for instance in the previous example on if to give 100000-300000 positions/level on the main signal, and on an impulse 12000000-16000000 positions, then after scaling there will be 65000bits(positions) on an impulse, and nearby 1200-1500 on the main signal. That is, as a result we see maximum detailing of random noise, and rough detailing of nessecary content.

SA preview mode of waveform in the loader, gives possibility to scale a signal before loading, that for the last example guarantees effective allocation of all dynamic range on the main signal, by simple cutting off of everything that exceeds a threshold, as a rule it can be various artefacts or powerful impulse interferences.

This SA possibility is entered for 24 and 32 bit records formats, including the format with a floating point, as 8 and 16 bit data are usually already brought to the necessary scale in advance, and if level of such records(24 and 32 bit) is very small initially, then scaling and amplification will not change situation to the best.

16 bits in overwhelming majority of cases is more than the sufficient value of digitization on level. If to follow a simple rule, then the signal should use as much as possible completely all dynamic range. Complete maximum does not mean from a maximum to a minimum, quite reasonable value is 70 %-80 % of a complete range, the small store always needs to be left, it concerns not only to 16 bit records, but any other format.

Usually the problems with 16 bit digitizations appear, when bars in 10-100 Mhz are inserted in record , in hope after to cut the necessary segments in tens Khz. It is an error. As to SA, SA iscertainly universal analyzer, and allows to make a lot with record, but it is not off-linereceiver, and to assign to it these functions at least is not recommended. SA will perfectly work out with 50 Mhz and above if the signal has bar at ~ 10-20 Mhz, but we cannot guarantee anything if someone intends to cut and analyze signals at 5-10 or 100 Hhz from 50 Mhz. All that should be done at the stage of reception of a signal, that is in RX. Basically, this is is one of the primary goals of the receiver, which for some reason frequently mistakenly supposed to be done by analyzers.

Actually these problems are not such big secret. The huge percent of complete not liquid records walks in the net from one site to another, I with some surprise detected "new" resources where represented "examples" of the various signals, and these "samples" we saw 5-6 years ago, on complenetly another resources.

Good luck!

New article: I/Q format. SA and I/Q records.

I/Q format records. SA and I/Q records.

Author: SergUA6.

We have received some letters, concerning I/Q format and usage of I/Q records in SA.

Let's try understand, what is I/Q record, why is it good and what possibilities it represents for the analysis and signal’s identification.

Why is I/Q format good and what possibilities it represents?

Standardly, 90-80 % of usual receivers, produce, as result of their operation, an audiostream. It is the real signal, which is standardly written in wav file, and after that, it can be used for analysis and identification, or placed somewhere on the Internet. Quality of such signal is completely depends on receiver’s characteristics.

The receiver in this case, is fully realizing processing of the signal, and in most cases, receiver "solves" for you, what and how to do with the signal. Of course, it is supposed, that the receiver is managed/controlled/used by competent user, however, practice shows that it is far not so. High enough number of people, still, do not suppose, what is going on inside of receiver, and what is going on at the recording process. That is why Internet is full of records with the very low quality, or even "non-liquid" for analysis/identification record. In most of such records, we do not have the real signal, but something which is only a bit similar to it, or even not similar at all, in very bad cases. The main problem, is that, far not each user wants to gain an understanding of the very big diversity of various classes of signals and special sorts of modulations/manipulations.

Luckily, today, old enough and known ideas, thanks to existing computer technologies and its power, are successfully realized in the form of SDR technologies. What is SDR technology? SDR technology means, linear qualitative transfer of required segment of a spectrum, into area of low frequencies, and its further processing.

All charm of this technology, is that for this technology it doesn't matter what sort of modulation in the signal, and what are features of the signal, how high the skill of concrete user - the signal will be transferred on relatively low frequencies without distortions, in it's initial form. After that, the signal can be saved for the detailed analysis.

Please, pay attention, that we are speaking not about the signal after demodulation and processing, as it becomes in usual receivers, but about the signal in that sort as it is. In what sort the signal is represented in this case? It is I/Q format. Sense and method of I/Q format forming is represented below:

The signal (necessary range) :

1) is selected, if it is necessary, by the entry bandpass filter

2) and it is transferred on zero IF in the form of two quadrature channels I and Q

3)then each channel is filtered by LPF filter and moved on ADC.

As a result of such transferring, in each channel, there is a folded up/convoluted half-and-half spectrum of the main signal and phase shift between channels is equal exactly 90 degrees.

(Let's not go deep into Mathematics, all this is well described in the theory).

Such approach allows:

in the first, to reduce sampling rate in each channel, at the expense of spectrum's convolution

in the second, to develop the signal into "complete growth" at any time, in that sort, in which it was, when it is necessary, at the expense of phase shift in 90 degrees between channels.

If digit capacity of ADC is selected correctly, and own level of noise of conversions is small, then having saved channels I and Q in the record form, we are getting an absolutely exact initial signal, without any distortions.

Standardly, for these purposes the stereo format of wav files is used. In one of channels samples

I are written, and in another one samples Q, or on the contrary, it does not play a special role.

Thus, the record of format I/Q is the standard stereo file, which demands special processing further.

How to work with I/Q records in SA? Easy!

SA it positioned as software for the qualitative analysis (signals correct identification), and, of course, I/Q format takes a special place in SA, because I/Q format is potentially guarantying an absolute quality of initial material.

That is why, there is special mechanism in SA, which allows to receive the real, present signal from I/Q record, for the further operation with this signal.

So how it looks on practice? It is very simple. At loading of standard I/Q record, SA is warning you, that the record is the stereo record, and SA suggests :

to select the channel

to load all samples

You need only select the point "load all samples", and converting of I/Q record into the real signal will begun.

That's it! Now, it is possible to work with the signal in complete volume of SA possibilities. You can save the file, filter it, re-sample, demodulate it and etc etc etc.

It is noticeable that the sampling rate of the file with the real signal is twice more than initial I/Q, and it is normal. Because now the signal is presented completely as the real one, and instead of two channels as earlier, we got one. However, data volume is remained precisely same: data is not reduced or increased in the file with real signal, the data in such file is simply represented in other sort, which is more convenient for human.

Why some people are still deny I/Q format, while it is obvious that I/Q is the most suitable for analysis and identification?

It is obvious that format of I/Q records is the most preferable for analysis, and it seems like it is clear enough. However, on practice, we often meet complete not understanding of this fact.

And there are some reasons for it. Let's consider them:

The first and the main reason, is that, almost all owners of SDR receivers, are confident, that various demodulators, which are included in software, which is supplied with these receivers are miracle.

Please do not get under self-deception, everything is not so is easy, and mostly, everything is not easy at all.

Necessity to work in real-time, orientation on radio signals listening, and many other factors force to use solutions, which are far from the ideal. In result, after demodulation, such records become nothing better, and sometimes even worse than, the records, which are done by usual receivers.

SDR receiver, which does not allow to write necessary bar or a signal in I/Q form, is lame by default

If you want to correctly identify any signal, first of all, You should be interested it the signal’s initial form only. And such form can be provided only in pure I/Q format.

Some people, who already faced with I/Q files in SA, have noticed that, sometimes, level of I/Q record is very low.

It is not the problem of SA. It is the problem of SDR receivers and their circuitry.

Maximum, what it is possible recommend in such situations, is to try to increase amplitude of a signal as soon as possible, before any manipulations with the file.

For example, after loading I/Q record, we can, at once, estimate and increase signal level if it is necessary:

Then it is necessary to save the file as usual wav, and after it is loaded again to process it as I/Q. It can compensate weak level of initial file in some measure.

Although it will not completely solve this problem, I underline, this is not SA problem, and this problem should be solved on the stage of the record's forming, because these are questions of SDR equipment competence.

The second reason why some people still do not consider I/Q as the most suitable for analysis format is that: there is an opinion that I/Q record is too plentiful and takes unfairly big volume. It not so truly.

I/Q assumes, that signal is transferred on zero frequency and if, to write only the signal with a reasonable store, then the volume/size of the record will be exactly such as necessary, no more and not less.

Another matter, if the signal has a bar in 10 kHz, but whole segment of range of the signal radiation (for example 1Mhrz) is fully transformed in I/Q, then of course, in this case, we have an absolutely redundant record. This kind of records says only, that the user who have done this record, simply do not understand he was doing.

Do not compare size of records of the signal after demodulation and the size I/Q record of a complete signal before demodulation.

In the first case, You work only with image of signal, and You do not know how was this image gotten, and if this image is even correct.

In the second case, You work really with the present signal, and only You solve when and how should it be demodulated.

Surely, in the second case, the chances of correct identification of the signal are incommensurably higher then in the first case.

It is very strange to hear that I/Q has too big unjustified volume from the people, who are making demodulated signals with bar in 3 KHz, in wav files with sampling rate 44100 Hz, in stereo format, and after that these people contrived to compress it by mp3 codec!

After all they are sincerely surprised that nobody have any interest in their records.:)

Record for the following analysis.

If you record the signal just for yourself, then you can do not read all that follows bellow. But if you are making
the record, specially, to represent it to public, or to give it for further analysis, then you must take care about
quality of this record.

Good quality of the record of the signal – is the half of success.

1. Never record with digit capacity of 8 bits. This 8bit mode suits to everything, but only not for finding-out of
nuances of the signal’s structure. Use only 16bit. That is not the point, that 8bit it is bad, the point is that
16bit, by default, provides the quality, which is enough for analysis and disassembling, while 8bit provides such
quality only in very rare cases. There is also no sense to convert the record from 8bit into 16bit artificially – it will be self-deception: all subtle connections inside the signal will be lost irretrievably.
Forget about recording from a microphone, it will be the fatal end of your record, don’t even not think to publish
it, only a linear output, or in extreme case, an output on ear-phones.

2. Never write the record in the compressed format at once. The record if it is really interesting to you, and in
some sense is precious, should be written in the pure form, without a compression – in the old kind WAV format.
underline, not some sort of WAV format(for example ADPCM and many others), but namely the pure PCM WAV standard. The original record should have the highest quality, then, if possible, it can be compressed, but if to compress it at once - an initial clear picture of the signal will be lost and not available for restoring any more. This is very bad because, there are lost of situations, when not compressed signal can be demanded easily.

3. It is necessary to approach with all gravity and seriousness to the record’s sampling rate - for a large number of signals the sampling rate should be not below then 11025 Hz, that provides the upper boundary of a range on the level about 5.5 kHz. For the record of satellite signals it should not be lower then 22050 Hz.
Do not hope that it is enough 8kHz sampling rate for telephone channel, all this is true, but in terms of monitoring,
particularly on HF, the inaccuracy of the settings and features of the modulation can easily deflect the signal from
the telephone channel and the "unnecessary" half kilohertz is not only interference, but the guarantee of deflection.
Only after the record is done, it is possible to reduce the sampling rate, by results, if it is required. The records
of satellites are, generally, easily overlapping upper boundary for 6-8 kHz , thus, remember - the sampling rate for them should not be less than 22050 Hz!

4. Never make the record without preliminary check-up/setup(setting), of path(tract,route) of the record. There is nothing worse, than the rare record, which is made with an overload.
Such records do not cause anything but regrets, it is impossible to understand them, to result them into normal kind is also impossible, you can only throw them out into basket.
Pair of the minutes, which are have been saved by the lack of knowledge or by the laziness, will expressed in a waste of time and an absolutely useless resources.
The maximum signal’s level at the recording should not exceed half of the possible maximum, thus guaranteed the
existence of at least some reserve, in case of any surprises.
It is better to write with certainly reduced level, than with certainly high.
Weak level can be raised, while the overload in the record cannot be removed by any weakening , the signal’s
form/shape will be distorted to unrecognizability.

5. If you turn the receiver with purpose to write something "special", then immediately turn on the record. There are a lot programs, which are writing the record into long file, personally I use RecAll, it writes, while I’m busy withmonitoring. The point is that the rare and interesting moments are always occur unexpectedly, and they are very short, the vanity, which is arising from convulsive attempts to launch /run/start recording program, leads to the fact, that neither the record nor the signal is there in the end. When the record goes permanently, the risk to skip something interesting is about zero, it is necessary only to be provided with reasonable time of the record into "a ring" file, for me this reasonable time is about an hour.

6. Do not do anything with the record, do not improve it and do not try to make it look better, leave it fully "as it
is". You can easily corrupt it, and what you seem as " unnecessary" can play the very important role in the analysis.

7. Okey, so the record is have written, and everything seems good to you, you want to public this record for an
identification or for other purposes – please be careful, do not create illusions, view the record by any editor or a
software, and check what is exactly you got in that record.At least estimate the form of the signal: there should not be any restrictions by the amplitude or any unexplained strangenesses. If you do not understand what is happened within your record, then others, most probably, won’t want to understand it especially.

And in conclusion: To make a good record - it is same work and skill as all other professional skills, nothing can
be well-done by itself. Train, try, play in quiet conditions. Look what is overload on practice, check what reasons
cause overload, try to remove it, learn how to define it.
It is not enough just to have a knowledge, it is necessary to know how to apply this knowledge, the only one method to know it – is the practice and experience.

Author SergUA6