texts: can someone give me an idea how to extract information for text such as invoice bills using natural language processing

texts: suppose i have text file with invoice details in it . I want to extract only some information based on my certain condition such as Mobile - 25,000 and quantity - 1 i want to extract only mobile and quantity based information and store it in text file.Can someone give me an idea how to do this.


texts: How does xgboost work with linear booster?

texts:
I know how to implement linear objective function and linear boosts in XGBoost. My concrete question is: when the algorithm it fits the residual (or the negative gradient) is it using one feature at each step (i.e. univariate model) or all features (multivariate model)?

Any reference to documentation about the linear boosts in XGBoost will be appreciated.

Thanks!

EDIT: Linear boosts can be implemented in XGBoost by setting the 'booster' parameter to 'gblinear'. See:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/
for useful information on linear boosting. Note that I am not speaking about the objective function (which can be also linear) but about the boosts themselve.


texts: Can Convolutional Neural Networks (CNN) be represented by a Mathematical formula?

texts:
Let's say that I already trained my CNN. Is there anyway of my ouput to be represented as a formula just like a perceptron can (x1w1 + x2w2 + ... = PREDICTION).

It does not matter if the formula is more complicated than the perceptron one, but in general would it be possible to train a CNN in Python or Matlab, get the weights and create an arithmetic, exponential, whatever formula made with those acquired weights?

With that, after training, I would be able to apply my new values to this formula and get a desired output/prediciton values.

I want to do this because I am trying classify in a PIC32 (a low cost microchip) which I think that does not have the capacity to perform a CNN within itself.

Or, for example, should I pursue the idea of some sort of "C library" that can do CNNs?


texts: Why is the numeric column treated as a categorical column in Microsoft Learning?

texts:
Learning Microsoft's DP-100 and saw this notebook:
The part of its code which I have question with is
# Define preprocessing for categorical features (encode the Age column)
categorical_features = [7]
categorical_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])

here is the link for the entire notebook for you to see the full picture. I cannot copy the full pages of code here.
Here is how the data look like:

My question is: Why is the DiabetesPedigree column(which is the 7th column) treated as a categorical feature?
To me, it looks nothing different from the rest of other columns(except the Diabetic column).
If DiabetesPedigree could be treated as categorical why not Pregnancies?
What are the rules to tell whether a column of data is numeric or categorical?


texts: How to handle potential interactions when one-hot encoding?

texts:
Let's say I have two categorical features: Movie, Director. I one-hot encode both the Movie and Director features for use in a linear regression model.

The problem is that two or more movies may be directed by the same director. i.e. a particular director's bit may be on for two or more different movies. Would this be a problem? Should I be combining movie and director into one feature?


texts: What's the best way to select features independent of the model being used?

texts: I am using tensorflow's DNNRegressor to model a multivariate regression problem. I want to form an optimal feature subset from a mixed bag of categorical and continuous features. What would be the best way to proceed? The reason I want this approach to be independent of the model is because I couldn't find much about feature selection/evaluation in direct context of tensorflow.


texts: Show product saldo by year from created/deleted date

texts:
I have a product database with a created and a deleted field in datetime format.
I want to aggregate this data into a bar graph with count(created)/year, count(deleted)/year and saldo/year.
I have already accomplished this for one year with calculated fields:
Created_2020:
IF DATEPART('year',[created]) = 2020 THEN
1
ELSE
0
END

Deleted_2020:
IF DATEPART('year',[deleted]) = 2020 THEN
-1
ELSE
0
END

Saldo_2020:
IF [Created_2020] + [Deleted_2020] != 0 THEN
[Created_2020] + [Deleted_2020]
END

How can I do this for multiple years?


texts: Has anyone applied Machine Learning methods to Geochemical Data?

texts:
I am looking to do a project on geochemical data , on which I will apply Machine Learning Technqiues to hopefully achieve good results.

I was wondering whether anyone has done anything like this and could guide me to some resources or research papers?


texts: How to calculate separate binary accuracy metric for each label in a multi-label classification in Keras?

texts:
I'm stuck with this:
def multilabel_binary_acc(y_true, y_pred):
    return [K.equal(y_true[:,n], K.round(y_pred[:,n])) for n in range(y_pred.shape[1])]

But it returns only a number similar to the binary accuracy metric from Keras.


texts: Calibrating probability thresholds for multiclass classification

texts:
I have built a network for the classification of three classes. The network consists of a CNN followed by two fully-connected layers. The CNN consists of convolutional layers, followed by batch normalization, a RELU activation, max pooling and drop out. The three classes are imbalanced (as can be seen in the confusion matrix below). I have optimized the parameters of the network to maximize AUC.
I'm calculating the AUC using macro- and micro-averaging. As can be seen in the ROC plot, the AUC is not that bad. On the other hand, the confusion matrix looks pretty bad, especially the first (low) class is badly predicted. The network tends to predict the majority class. As output of the network I'm getting a probability for each class. Then, I'm just taking the class according to the maximum probability for creating the confusion matrix.
I have tried to use balanced class weights while training the network (in the fit method of Keras). This helped that the network also predicts more often the minority class(es) but on the other hand the AUC was decreasing.
Is there a way to infer probability thresholds from the ROC plot? I think for two classes the optimal probability threshold can be inferred from the ROC plot by taking the max(TPR - FPR) but here I have three classes... Or is there another method?


texts: Error: ValueError('%r cannot be used to seed a numpy.random.RandomState')

texts:
I am getting this error message while trying to fit a model for the isolationForest algorithm.

raise ValueError('%r cannot be used to seed a numpy.random.RandomState'

Below is my code:

 import numpy as np
 import matplotlib.pyplot as plt
 from sklearn.ensemble import IsolationForest
 import pandas as pd

np.random.RandomState(1234)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
df = pd.read_csv('E://Market_dat.csv',names=['EVENT_DT', 'MARKET_NAME', 'Duration', 'TOTAL_COUNTS'],skiprows=1,index_col=0)
for column in df.columns:
if df[column].dtype == type(object):
    le = LabelEncoder()
    df[column] = le.fit_transform(df[column])
    np.random.get_state()

X_train, X_test = train_test_split(df, test_size=0.3)
print(X_test)
print(X_train)
X_outliers = rng.uniform(low=-4, high=4, size=(20, 2))
clf = IsolationForest(behaviour='new', max_samples=100,
                      random_state=df, contamination='auto')
clf.fit(X_train)

Can anyone give any insight as to why I might be getting this error?


texts: How to decide to use K-fold validation or not?

texts:
When K-fold validation should be used and how to decide the value of the K.
I see most of the text books suggesting implementation of algorithms using 2/3 of data as train and 1/3 of data as test


texts: Combining time-series data from different devices

texts:
I am using four different sensor models (four different devices) to gather  some biometric data(eg. heart rate) for the same user, for the same time period. In an ideal scenario, all the devices are properly positioned, so for the time being we are not concerned about null values and bad readings.

The data follows this format:

timestamp: XXXXXXXXXX //unix epoch timestamp
duration: XXX //duration in seconds
value: XX
state: <state> //string that characterizes the state of the user based on the value

The devices have quite varied levels of sensitivity, and their individual graphs may contain different "state" sequences, both in terms of the number of appearances of each "state" and their individual timestamps.

I am now tasked with presenting the data into a single graph that combines the data from all devices, and clearly presents the "state" of the user over time. Is there a way to do this without making arbitrary decisions regarding the accuraccy of the devices, or the significance of small duration readings? (I am doing this in Python if that is of any importance)


texts:
Let's say I have images of cars. For each image in the dataset, I have let's say 3 pictures of the same car but in different angles.

1) The first image is the picture of the car from the front.
2) The second image is the picture of that car from either of the sides.
3) The third image is the picture of that car from the back.

If I have 1000 cars, I would have 3000 images as my dataset. Is there a way to find which angle of the car represents its label the most? In other terms, Which angle of the car image is most helpful in determining whether a particular image is a car or not?

My thoughts were to build a random guessing model to determine the lower bound of our loss and then iteratively use each of angle of images as train and predict on the same angle and come up with a loss score. Ex: First I train and predict with all images of angle 1 and come up with a loss score. So If I have 3 angles of images, I would end up with 3 loss values. So the angle with the lowest loss score is determined as the winner. But the problem is, I'm not sure if this is approach is right because if I have multiple classes in the dataset then the dominant class would have lower loss compared to any other non-dominant classes and moreover we would not be using the entire data at any point of time and so would possibly end up with wrong conclusions.

Thanks for replying in advance. This question has been lingering in my head for a long time now.


texts: Adding layer to a trained CNN to process higher resolution images. Tried 2 schemes, 1 works fine, 1 fails completely

texts:
I'm working with images coming from a sensor, for which 1 pixel corresponds to 2 mm in the real world. I've built and trained a CNN that does semantic segmentation of the image (128x128 pixels) and it works quite well. The objects that need to be recognized have a specific dimension in mm with little variability. So, if the pixel size is changed, the number of pixels taken on average by an object changes. It is not optimal to use a network trained with 2 mm pixels and feed it a patch of an image with 1 mm pixels.
Now we have a new instrument with higher resolution, 1 mm pixel size (256x256 image size). The higher resolution images are less, the training would be more complicated with many more free parameters. Also I'd like anyway to keep using my CNN that works well. So I've looked into transfer learning. However
Scheme 1 (fails) . Add a couple of convolutional layers at 1 mm, a downsample (average pooling) layer to 2 mm that "injects" data directly in the pretrained network (right after what would be the input convolution/activation). Then I freeze all pre-trained layers and let the networks train. It's so trivial it should work. Instead it doesn't. I've even tried changing the loss and making it so that the output of the first 2 mm convolution should be identical (mean absolute error between the output of the network with the 1 mm input and the network with 2 mm input given the same input image downsampled). By definition it should reach 0 loss in a couple of epochs but it doesn't! The output gets very similar... but small differences then explode in later layers and the output is nowhere similar to what it should be. Like dice 0 instead of 0.9, and visibly is totally wrong.
Scheme 2 (works well) . My 1 mm input gets directly fed in an average pooling layer and then fed in the 2 mm network. In parallel it goes into a couple of convolutional layers. Then the last layer of the 2mm network (before the 1x1 final convolution) gets upsampled to 1 mm and concatenated to the 1 mm layer. This works perfectly without resorting to any special trick for training.
So... Why does my scheme 1 fail?? Any theoretical issue?


texts: Understanding Youtube recommender (candidate generation step)

texts:
I'm trying to understand Deep Neural Networks for YouTube Recommendations.
Their candidate generation step outputs top N items

via softmax (with negative sampling) at training time .
via nearestneighbor at serving time.

I guess $v_j$ represents, (from softmax layer to nearest neighbor index)
topn videos you get via softmax, and represent them in the original encoding (same encoding you used for the input (used for embedded video watches))
apparently, $v_j$ are in the different encoding from the input encodings.
The softmax layer outputs a multinomial distribution over the same 1M video classes with a dimension of 256 (which can be thought of as a separate output video embedding)
I'm trying to understand what they mean by interpreting softmax output as a separate output video embedding.  I thought softmax layer that outputs 1M classes has dimension of 1M, where does 256 came from?  (It's the same question as How to create a multi-dimensional softmax output in Tensorflow? and I don't think it has been answered there..)

user vector $u$ is the output of the final ReLU unit, although I'm not sure what this user vector is used for.

I guess in serving time, to pick the top N for a given user, user vector $u$ is used by nearest-neighbor. But my understanding of nearest-neighbor is for a given vector, it finds nearest vectors in the same dimension. (such as given an movie, find nearest movies). However here, you are given a user and need to find topn videos. How does that work?
My best guess is that, for a given user, u get a user vector as the ReLU output, then find user-user nearest neighbor, and combine their topn items obtained in the training time. But it's just a guess..


texts:
I am fitting a Keras model, using SGD
Input dataset X_train has 55000 entries.

Can anyone explain the yellow highlighted values?
For me, when each epoch is done, this should correspond to 55000/55000.
model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28,28]))
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dense(100, activation="relu"))
model.add(keras.layers.Dense(10, activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy", optimizer="sgd",metrics=["accuracy"])

history = model.fit(X_train , y_train, epochs=30 , validation_data=(X_valid, y_valid))


texts: r - How to determine the correlation between unordered categorical variables and individuals?

texts:
I have a matrix with several unordered categorical variables. Each row represents a type of individual. Each column represents the number of times each type of individual was found to be in that particular condition.

Type    coal    cobalt  concrete    copper  gold
A       12      0       0           19      5
B       5       0       0           11      0
C       4       2       0           14      1
D       1       3       15          0       1
E       0       20      2           1       9

My question is very simple: I want to know if there is a correlation between the type of the individual (A, B or C) with a particular condition (copper, gold, etc).

Which test should I use? If possible, I would like to get the answer by using R.

Thanks!


texts:
I work with Keras for two (or so) years now and I have approached a number of different regression and classification problems using Keras.

I recognized that my models sometimes learn nothing, meaning that they cannot beat a naive Bassline (e.g. 50% accuracy in binary classification).

Note that the same (or very similar) model did learn much (!) better in previous runs or when I restart the learning process. So there seems to be quite a random element in each model training run.

Does anyone know where these effect comes from?


texts: Finding a proper method for an online driving style classification using acceleration data

texts:
I am using a smartphone in my car to gather acceleration data (both longitudinal and lateral). Now, I want to classify my data in real-time based on the acceleration force applied through accelerating, breaking and maybe turning.

Is there a proper method to do this kind of online classification?
And also how does one determine whether a "harsh/aggressive" or just "normal" driving style?


texts:
I'd like to build something (ideally in Python) that can translate an English sentence into "basic" English.
Are there any free/open-source tools/frameworks that can help? If not, what kind of steps can help solve this problem? (e.g., building on existing work like WordNet or pre-trained word embeddings).
By "basic" I mean things like being concise (avoiding unnecessary verbiage) and using well-known words (without compromising too much on meaning). I might even consider "broken" English, where verbs for example are lemmatised to their root.
The idea is to get something akin to what Wiki have achieved with their basic English versions of articles. And to give one more example, simplifying sentences is one of the services provided by the Grammarly app.

For example:
English:
I hurriedly made my way over to the store in order to purchase various garments from there

"Basic" English
I quickly went to the shop to buy clothes

"Broken" English
I quick go shop buy clothes


texts: Comparing multi-class results with binary classification results

texts:
We used machine learning to discriminate the following five disease classes:

Normal (N)
Myocardial Infarction (MI)
Coronary Artery Disease (CAD)
Congestive Heart Failure (CHF).

In the past, these diseases were only discriminated with binary classifiers: N and MI, N and CAD, N and CHF.

What is the best way of comparing the multi-class results with the three individual binary classification results?

Access to the confusion matrix is available for all cases.


texts: Predicting Y Values Properly in a Regression Task using Scaled Values (Random Forest & MLP)

texts:
I have a supervised learning regression task: I am trying to forecast demand for a product based on sales in past years.
Data description:
Samples (rows) - Demand for a certain product (at a certain month, in a certain year). Number of rows: 2,400
X Variables (columns) - Explanatory variables. Most of them are categorical (which I turned into dummy variables), 2 of them are continuous ("Price Per Unit", "GDP"). Number of X variables: ~20
Y Variable (target): Demand in units.
I decided to use Random Forest Regression and MLP Regression for the task, using Python & scikit-learn.
For both models, I used StandardScalar to scale the continuous values, and then used RandomSearch for hyper-parameter tuning to find the optimal model.
My question:
After finding the optimal parameters, do I need to fit the model again (with the same optimal parameters) but on the original, un-scaled values? (Is the scaling meant to be only for FINDING the optimal parameters?)
Also, when predicting values for new data: do I need to scale the values again, or do I just use the original, un-scaled data? (If I scale them, the model will return scaled Y predictions, which I cannot interpret as a solid demand in meters...)


texts: An abstract idea for the performance diffs between SLP and MLP

texts:
Recently I am working on some predictive analytic which based on neural network.

When I tried some tests on MLP with one hidden layer or multiple hidden layers, the results showed that:

one hidden layer performance on prediction is always better than multiple hidden layers
the tests were executed both on Knime and R models, they gave me the same trends

Based on my knowledge, more hidden layers with more neurons should perform better?

Is there a principle on this about for which kind of dataset turns out this kind of result?

Or I may need some book / article / paper to read?

Do you have any abstract idea (not mathematical algorithms) for me, thanks!

UPDATE

I am working on a fraud detection dataset, which includes 1000 observations. and all the dimensions are numericed and normalized...

But I am actually asking for a general idea, for the general idea to choose algorithm for different dataset


texts: High Training Accuracy, Poor Validation, Test Accuracy

texts:
I am a beginner exploring Deep learning. I am trying to train a classifier (9 classes) with images as the input to my CNN followed by Bidirectional LSTM architecture.

My model rapidly achieves a very high training accuracy (98-100%) but the test and validation accuracy remains constant at 15-20%.

In the above image, blue line is the training accuracy. Green is validation accuracy and yellow is test accuracy

In the above image, the blue line is the loss which is being propagated backwards

What could be the possible reason for this peculiarity? Any suggestions to overcome this poor test accuracy dilemma would be appreciated.

Additional Info: I have 630 training samples, 90 test samples and 90 samples for validation. I have implemented early stopping to stop training once my validation error stops decreasing in a patience interval.

# Convolutional Neural Network

class ConvNet(nn.Module):
def __init__(self,num_classes=9):
    super(ConvNet,self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(in_channels=3,out_channels=16,kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2,stride=2),
        nn.Dropout(0.25)#dropout_factor = 0.1
    )

    self.layer2 = nn.Sequential(
        nn.Conv2d(in_channels=16,out_channels=32,kernel_size=3,stride=1, padding=1),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2,stride=2),
        nn.Dropout(0.25)#dropout_factor = 0.1
    )

    self.layer3 = nn.Sequential(
        nn.Conv2d(in_channels=32,out_channels=8,kernel_size=2,stride=1, padding=1),
        nn.BatchNorm2d(8),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2,stride=2),
        nn.Dropout(0.25)#dropout_factor = 0.1
    )

    self.flc = nn.Linear(8*16*32,1024)

    self.hidden_size = 9
    self.num_layers = 2

    self.blstm = nn.LSTM(input_size=1024,hidden_size=9,num_layers=2,batch_first=True, bidirectional=True)

    self.flc2 = nn.Linear(18,9) #(hidden_size*2, num_classes) %2 for bidirection

def forward(self,x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    out = out.reshape(out.size(0),1,4096) #Flatten tensor
    out = self.flc(out)

    h0 = torch.zeros(self.num_layers*2, out.size(0), self.hidden_size) # 2 for bidirection
    c0 = torch.zeros(self.num_layers*2, out.size(0), self.hidden_size)

    # Forward propagate LSTM
    out, _ = self.blstm(out, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size*2)
    # Decode the hidden state of the last time step
    out = self.flc2(out[:, -1, :])
    return out


texts: Soft and hardware for manipulating multi-million record data sets

texts: If starting from scratch, what hardware and software would be ideal and under \$5k or \$10k for manipulating a data set of 10 to 20 million records without slowing to the crashing point or taking several hours to complete? Manipulation, if it matters, means multivariate regression (simple OLS with less than a dozen variables), basic graphs, and tables.


texts: Would a simply Python Script to extract Information from a Webpage (e.g with use of beautiful soup for example) be called a Wrapper (see https://en.wikipedia.org/wiki/Wrapper_(data_mining))?


texts:
I'm confused about input data on LSTM neurons. I know that exist almost two form to give data to a recurrent neural network.

I want to understand with a example

Data
[1,2,3,4,5,6]

For feeda RNN, data must be formatted with this schema:

(x,y,z)
    x = samples
    y = timestep
    z = features

But, I know three ways to input data:

Way 1
Shape =  (x,1,3)

data = [
        [[1,2,3]],
        [[2,3,4]],
        [[3,4,5]],
        [[4,5,6]],
        ]

Way 2
Shape = (x,3,1)

data = [
        [[1],[2],[3]],
        [[2],[3],[4]],
        [[3],[4],[5]],
        [[4],[5],[6]],
        ]

Way 3
Shape = (x,2,2)

data = [
        [[1,2],[3,4]],
        [[2,3],[4,5]],
        [[3,4],[5,6]],
        ]

If input is way 1, I supose  LSTM neurons = z.

But I don't know what receive the neurons in way 2 and 3.


texts: Keras CNN always predicts the same output values, independet of the input

texts:
We train a Reinforcement Learning model that should always predict the next best action in each state.
Somehow the training does not work since after a few games, the outputs for each of the actions converge to specific values. So in other words, the model predicts the same probabilities for each action, indepentent of the input. This is our model:
q_net = models.Sequential()
q_net.add(Conv2D(16, (5, 5), activation='sigmoid', input_shape=(17,17,1)))
q_net.add(MaxPooling2D((2, 2)))
q_net.add(Conv2D(32, (3, 3), activation='sigmoid'))
q_net.add(Flatten())
q_net.add(Dense(128, activation='sigmoid', kernel_initializer='he_uniform'))
q_net.add(Dense(64, activation='sigmoid', kernel_initializer='he_uniform'))
q_net.add(Dense(32, activation='sigmoid', kernel_initializer='he_uniform'))
q_net.add(Dense(6, activation='sigmoid', kernel_initializer='he_uniform'))

q_net.compile(optimizer=Adam(learning_rate=0.0001), loss="mse")
return q_net

Can someone help me?
Thanks!


texts:
So have this table above. I'm trying to aggregate the occupations such that the table results in:

I've tried using df.groupby(['Occupation']) but I get an error.
All I know is that my final step would be to set the index to "Occupation". But I still don't know how to group via entries in the single Occupation column here.
Also, what type of table would the final table be name/called?
I know it's not called a mutiindex table because there is only one index that the results are being grouped by.


texts: What is the algorithm/logic top e-commerce companies use to show products in "What's new?" and "Bestsellers" sections on the website?

texts:
I am trying to solve this problem for a fashion e-commerce website that sells t-shirts, joggers etc.
Till now I am only able to make some rule based logics only. It looks something like this

The product must be launched in last 30 days.
The product must have sufficient inventory, so sufficient customers can place order.
Finally, sorting them according to average sales of last 7 days from high to low.

Links, references or approach to solve this problem would be very helpful.


texts: Which MSE (total or individual) back-propagate for multi out regression neural network

texts: When we have multi output regression neural network, we can calculate total MSE and individual MSE per output. How this MSE should back-propagate ? Shouldn't we back-propagate individual MSE through the network for relevant weight.


texts: Keras: matching logistic regression performance with sequential neural network?

texts:
Assume a binary classification problem and a relatively small dataset ($\sim \mathbb{R}^{5000 \times 39}$). By using common ML techniques, starting with logistic regression, I'm able to reach ~0.76-0.79 AUC on the validation sample, depending on the model. So there clearly are some significant signals to be extracted from the model.

However, when trying even a simplest sequential neural network, it does not learn at all.

I.e., using keras from R: (NB: AFAIK the language shouldn't be a problem, since it's only a wrapper for R)

model <- keras_model_sequential() %>%
         layer_dense(units = 48,
                     activation = "sigmoid",
                     input_shape = 39) %>%
         layer_dense(units = 48, activation = "sigmoid") %>%
         layer_dense(units = 2, activation = "sigmoid")

model %>% compile(
  optimizer = "rmsprop",
  loss = "categorical_crossentropy",
  metrics = c("accuracy")
)

history <- model %>% keras::fit(
  x_train,
  y_train,
  epochs = 500,
  batch_size = 16,
  validation_split = 0.7, shuffle = T
)

However, whatever number of epochs or batch sizes, the model doesn't learn at all, the validation accuracy corresponds to the distribution of the target variable (predicts all as 1).

What am I doing wrong? Shouldn't a small one/two layer NN with sigmoid activations at least reach the performance of logistic regression? Any suggestions of how to at least replicate the logistic regression results? Or maybe I've missed some steps in the code?

Would be greateful for any insight!


texts: Improving Training accuracy of LSTM in Keras for ratings prediction given reviews

texts:
I'm new to Deep Learning and NLP. I found a dataset online which has reviews of different companies and their corresponding ratings from 1 to 5. I encoded the labels, then removed some basic stop words from the text, and tokenized it all using Keras. My model, however, is not performing well at all. Training accuracy starts off at 30% and only goes up to 60%. Validation accuracy also starts off low, at around 35%, and ends at 30%. What do I do to improve the performance?
model = tf.keras.Sequential([
tf.keras.layers.Embedding(10000, 20, input_length=150),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.LSTM(40, dropout=0.2, recurrent_dropout=0.2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dense(15, activation='relu'),
tf.keras.layers.Dense(6, activation='softmax')])

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(padded, train_label1, epochs=10, batch_size=45 ,validation_split=0.1)


texts: Normalizing Feature/Label with Negative Values

texts:
I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object MinMaxScaler. However, one of the features, altitude change, contains both positive and negative values. In addition, the label, energy consumption, can be negative due to the energy returned to the battery from regenerative braking. I need my model to respond differently to these negative values as opposed to the meaningless negative values of other features like temperature.

Is it best practice to instead normalize some/all features from -1 to 1? Or should I be standardizing all the features instead (even if they may not be normally distributed)?


texts: Making sense of blocks python package output

texts:
I've modified the Blocks tutorial, to train an MLP neural net with a dataset provided for an assignment in my ML course.

I'd like to evaluate the accuracy of the network with varying parameterization, but I am not sure how to obtain the accuracy in first place.

Inspecting the main_loop object useing dir() & vars(), I'm not coming across anything other than the test_cost_with_regularization.

It is possible to record means of different Theano variables via the monitor classes, so perhaps the answer lies there within?

Final Output

---------------------------------------------------------
TRAINING HAS BEEN FINISHED:
---------------------------------------------------------
Training status:
     batch_interrupt_received: False
     epoch_interrupt_received: False
     epoch_started: False
     epochs_done: 1
     iterations_done: 16
     received_first_batch: True
     resumed_from: None
     training_started: True
Log records from the iteration 16:
     test_cost_with_regularization: 2.3032071347
     training_finish_requested: True
     training_finished: True


texts: Why do we need activation function (like ReLU) after an affine layer?

texts:
In Convolutional Neural Networks, assume the input and the output of the affine layer are $x$ and $y$, respectively. This affine operation $y = W^{\top} x + b$ has already add non-linearity to the system given that $b \neq 0$.

Why do we still need a function like ReLU to add non-linearity to the system?


texts: Removing duplicate records before training

texts:
I am currently working on a project classifying text into classes. The specific problem is classifying job titles into various industry codes. For example "McDonalds Employee" might get classified to 11203 (there are a few hundred classes in the problem). For this we are using FastText.

The person that I am working with insists on removing duplicate records from the data before training our model. That is, we might see 100 records with "McDonalds Employee" and class 11203 and he wants to remove all but one of them. His argument is that not doing so could result in over-fitting and an optimistic error rate as the same records will appear in all the train/test/validation sets. My counter to this is that I expect to see (many) records with "McDonalds Employee" in our future data and I would like to know how the model is going to do at predicting this, hence we will not arrive at an optimistic error rate but a properly calculated error rate. Secondly, if our data for some reason has one record "McDonalds Employee" with class 24444, removing the duplicates removes all evidence that the correct code is 11203.

I have read other posts here that suggest removing duplicates is not correct, but I have yet to see an actual source in the literature stating this. Since I have to convince a colleague my question is two fold: Does anyone know of any reference in the literature that suggests keeping duplicates? As well, is there any reason to remove duplicates specific to FastText? I admit I am not that familiar with NLP and FastText (or even neural networks in general), so it is possible maybe there is some reason to remove them when training a model of this type.


texts: Is it possible retrive hourly data from daily weather data?

texts:
I'm a beginner, so sorry if my question could be basic. Reading on internet I've found example written in python that makes the reverse of my questions (covert from daily to monthly).
My problem is that. I've develop an ANN which requires hourly weather data (temperature, humidity, precipitation).
However I've only daily weather data for each day. (averange temperature, averange humidity, averange precipitation).
Is it possible retrieve from daily data the hourly data in python? if yes, how?which libraries?

Daily data

Data         station      avg Temperture     avg Humidity   avg rainfall
18/07/2019   Los Angeles  23                 50             0.0

I'd like to have that

Data         station       Hour   avg Temperture     avg Humidity   avg rainfall
18/07/2019   Los Angeles  0:00       23                 50             0.0
18/07/2019   Los Angeles  1:00       22                 45             0.0
18/07/2019   Los Angeles  2:00       20                 42             0.0
... and so on since have 24 intervals for each day

Thank you in advance


texts: Why does my model accuracy rise and then drop, with the loss sharing similar characteristics?

texts:
I'm working on a multiclass classification problem using Keras. I have nearly 17k training data, and approximately 14k testing data (validation comes from the 17k training data). I have a number of issues with my model which I would describe as follows:

Validation accuracy starts with a high of ~51% after three epochs and then starts decreasing to the point of after 100 epochs the validation accuracy is ~23%.
Validation loss value starts of at 6.2354, decreases for two epochs, increases for 10 epochs and then starts lowering again.
Training accuracy also decreases after 20 epochs reaching a high of ~40% and then slowly dropping with slight fluctations.

The above issues all occur while the loss value for the model is continually dropping. If anyone could help with this model it will be greatly appreciated.

In my code, I am each piece of data contains 4096 values each of which are floating point values, and I am trying to classify a class that comes in the form of a size 10 array that contains a number of 0s and 1s depending on the class that it is trying to classify.

I've also included a visual depictions of the way in which the loss value is behaving in the model for 100 epochs.

Code:

final_model = Sequential()

final_model.add(Dense(2048, activation='relu', input_shape=X.shape[1:]))
final_model.add(Dropout(0.5))

final_model.add(Dense(1024, activation='relu'))
final_model.add(Dropout(0.5))

final_model.add(Dense(1024, activation='relu'))
final_model.add(Dropout(0.5))

final_model.add(Dense(512, activation='relu'))
final_model.add(Dropout(0.5))

final_model.add(Dense(10, activation='softmax'))

cust = RMSprop(lr=0.00001, rho=0.9)

final_model.compile(optimizer=cust, loss='categorical_crossentropy', metrics=['categorical_accuracy'])

history = final_model.fit(X, y, epochs=100, batch_size=128, verbose=2, validation_split=0.2)


texts: Python: ggplot geom_histogram() binwidth or bin parameter missing?

texts:
I am unable to change number of bins when using geom_histogram() function. There is nothing in the documention, except some examples where binwidth=X is used (which is how R:ggplot2 uses it). However, this parameter seems to have no effect on the actual output.

Version: ggplot 0.10.4, Python 3.5 64bit

So for example:

import matplotlib.pyplot as plt
plt.figure()
dataframe['column'].plot.hist(bins=30)
plt.show()

works as expected and draws 30 bins. Meanwhile:

from ggplot import *
gg = ggplot(dataframe, aes(x='column')) + geom_histogram(binwidth=300)
print(gg)

does not change number of bins (I understand that binwidth sets interval for one bin), and have tried multiple values and/or datasets


texts: Training set of a NN that classify Cat or Not Cat

texts:
Is it possible to build a Convolutional Neural Network (using Keras, Tensorflow) that can give output as 1 for an image of a Cat and 0 for everything else?
How would the training set look like?
I mean, let's assume if the number of images of the Cat is ~1 million, then the number of images of "not a Cat" should be potentially trillion or even more.
More importantly, what classes of images should I include in the images that are labeled as "not a Cat", e.g. Dogs, Tigers, Zebras, Trees, Computers, Galaxies, etc?
Doesn't it mean that we should have an infinite number of training samples that represent "not a Cat"?


texts: Is it valid to add MAPE as a margin to prediction output?

texts:
I've trained a KNNRegressor on predicting used car prices. A given car's actual selling price is R289,995. My model predicts R260,911. I want to be able to tell the user My knn model predicts the deal is +-10% cheaper than the actual selling price.
To accurately present this to a user and to account for the model's chance of error, would using the mean error percent (14%) in this case, be a decent way to create an approximate range? Or is this not statistically valid.
So would it be more valid to say: Prediction range: R224,000- R297,000


texts: Prediction Algorithm for Data with high Randomness

texts:
I have data for the orders of the previous year containing the product and the seller who sold the product. I have an information product, product category, seller, delivery address price etc. Database size is more than 100,000 rows. Now, I have to suggest a seller for a totally new product based on the data I have. I tried using TF-IDF to find similar products of the same category to suggest the sellers and I got an accuracy of 70%. Then, I tried a random forest algorithm and sadly I got an accuracy of just 40%. I am unable to find a suitable approach for my use case. How can I approach this problem statement?
The Product and Seller Mapping table is like this

productId
sellerId
price
purchase Date
deliveryAddressId

1
4
100
9-01-2012
4

2
12
400
1-08-2020
4

1
1
123
4-09-2020
1

2
3
450
24-12-2020
1

3
4
150
14-05-2020
2

5
3
430
12-02-2020
2

Product has the following information

productId
name
categoryId

1
AC
1

2
TV
1

3
Food
2

4
Toy
3

5
Car
3

6
Book
4

Seller has the following information

sellerId
sellerName
totalTransactions

1
A
81

2
B
111

3
C
200

4
D
42


texts: Identify given patterns in unstructured data like text files

texts:
I wasn't sure if I had to ask it here or in Stackoverflow, but since I am also seeking research papers/algorithms and not only code, I decided to do it here.

When I have a text, I can manually write a regex to find all the possible outputs from what I want to extract from the file. What I want to do, is to find an algorithm or a research, which can let you highlight (set the input) different positions of the same (repeated) data you want to extract in the text file, train the algorithm and then identify all the others under the same contentions of those you set.

For example, let's say that I have a text with several titles which are following with \n\n\n and starting with \n\n. It is easy with regex, but I want to do it dynamically.

An idea is to build an algorithm which will take examples and create regex automatically. But I am not aware of any research like this and maybe there are also other techniques that you can achieve it.

Any ideas?


texts: How can I clean a data csv file with the restriction of only using python and its standard library? No third party programmes such as pandas can be used. For example: removing a column from the dataset, correcting spelling mistakes, inconsistencies in data formatting, null entries etc.


texts: How to detect if one tweet is agreeing with another

texts: I want to detect tweet text agreement. Suppose someone posts some subjective opinion in twitter. Other users will post reply either agreeing or opposing the original tweet. I want to estimate the amount of agreement. Is there any algorithm/library in any language to do that or any labeled dataset?


texts: What methods exist for distance calculation in clustering? when we should use each of them?

texts:
What methods exist for distance calculation in clustering? like Manhattan, Euclidean, etc.?
Plus, I don't know when I should use them. I always use Euclidean distance.


texts:
I have a decent understanding of word embeddings (at its core, one can think of a word being converted into a vector of, say, 100 dimensions, and each dimension given a particular value... this allows to do math with the words, also it makes the training sets to be non-sparse...)

But today something came to my mind, what about punctuation symbols such as , . () ? ! ... ?

They do have a huge impact on the meaning of sentences and, like words, the position and context in which they are used is relevant.

So the question is, how should this be modeled? are pretrained sets like GloVe including punctuation symbols? should I simply remove punctuation symbols from text?


texts: Using sklearn knn imputation on a large dataset

texts:
I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer.
Trying this off the bat I hit memory problems, but I think I can solve this by chunking my dataset... I was hoping someone could confirm my method is sound and I haven't hit any gotchas.
The sklearn KNNImputer has a fit method and a transform method so I believe if I fit the imputer instance on the entire dataset, I could then in theory just go through the dataset in chunks of even, row by row, imputing all the missing values using the transform method and then reconstructing a newly imputed dataset.
I'm wondering if there's an issue with this method regarding the chunksize or is the transformation on each new row independent?
50% of the dataset rows are fully populated... would it be better in terms of computation to fit the imputer object on only this portion of the dataset?


texts: How does Batch normalization help optimization? Proof

texts:
I am reading the paper How Does Batch Normalization Help Optimization found here.
$\newcommand{\norm}[1]{\left\lVert#1\right\rVert}$
But I am having trouble understanding the proof of the paper. It's about proving the effect of BatchNorm on the Lipschitzness of the loss. For brevity, what I am having trouble with is deriving:
$\norm{\dfrac{\partial \hat{L}}{\partial y_{j}}}^2 = \Big( \dfrac{\gamma^2}{\sigma_j^2} \Big)$$\Bigg( \norm{\dfrac{\partial \hat{L}}{\partial z_{j}}}^2-\dfrac{1}{m}\Bigg< 1,\dfrac{\partial \hat{L}}{\partial z_{j}} \Bigg>^2 - \dfrac{1}{m}\Bigg<\dfrac{\partial \hat{L}}{\partial z_{j}},\hat{y_j} \Bigg>^2 \Bigg)$
from
$\dfrac{\partial \hat{L}}{\partial y_{j}^{(b)}} = \Big( \dfrac{\gamma}{m\sigma_j} \Big)$$\Bigg(m \dfrac{\partial \hat{L}}{\partial z_{j}^{(b)}}-\sum^m_{k=1}\dfrac{\partial \hat{L}}{\partial z_{j}^{(k)}}  - \hat{y_j}^{(b)} \sum^m_{k=1}\dfrac{\partial \hat{L}}{\partial z_{j}^{(k)}} \hat{y_j}^{(k)} \Bigg)$
-- Details
$\dfrac{\partial \hat{L}}{\partial y_j^{(b)}}=\dfrac{\gamma}{m\sigma_j}\Bigg(m\dfrac{\partial \hat{L}}{\partial z_j^{(b)}}-\sum^{m}_{k=1}\dfrac{\partial \hat{L}}{\partial z_j^{(k)}}-\hat{y_j}^{(b)}\sum^{m}_{k=1}\dfrac{\partial \hat{L}}{\partial z_j^{(k)}}\hat{y_j}^{(k)}\Bigg)$ (1)
$\dfrac{\partial \hat{L}}{\partial y_j}=\dfrac{\gamma}{m\sigma_j}\Bigg(m\dfrac{\partial \hat{L}}{\partial z_j}-1\Bigg< 1, \dfrac{\partial \hat{L}}{\partial z_j} \Bigg>-\hat{y_j}\Bigg< \dfrac{\partial \hat{L}}{\partial z_j}, \hat{y_j}\Bigg>\Bigg)$ (2)
first I can't fully understand how (1) to (2), since <> is inner product, should result of inner product be scalar value? Since the result of $\sum^{m}_{k=1}\dfrac{\partial \hat{L}}{\partial z_j^{(k)}}$ is vector, how this and $\Bigg< 1, \dfrac{\partial \hat{L}}{\partial z_j} \Bigg>$ be same?
let $\mu_g=\dfrac{1}{m} \Bigg<1, \dfrac{\partial \hat{L}}{\partial z_j} \Bigg>$ $\hat{y_j}$ is mean-zero and norm-$\sqrt{m}$
$\dfrac{\partial \hat{L}}{\partial y_j}=\dfrac{\gamma}{\sigma_j}\Bigg(\Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big)-\dfrac{1}{m}\hat{y_j}\Bigg< \Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big), \hat{y_j}\Bigg>\Bigg)$ (3)
$=\dfrac{\gamma}{\sigma_j}\Bigg(\Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big)-\dfrac{\hat{y_j}}{\norm{\hat{y_j}}}\Bigg< \Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big), \dfrac{\hat{y_j}}{\norm{\hat{y_j}}}\Bigg>\Bigg)$ (4)
$\norm{\dfrac{\partial \hat{L}}{\partial y_j}}^2=\dfrac{\gamma^2}{\sigma_j^2}\norm{\Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big)-\dfrac{\hat{y_j}}{\norm{\hat{y_j}}}\Bigg< \Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big), \dfrac{\hat{y_j}}{\norm{\hat{y_j}}}\Bigg>}^2$ (5)
$\norm{\dfrac{\partial \hat{L}}{\partial y_j}}^2=\dfrac{\gamma^2}{\sigma_j^2}\Bigg(\norm{\Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big)}^2-\Bigg< \Big( \dfrac{\partial \hat{L}}{\partial z_j}-1\mu_g \Big), \dfrac{\hat{y_j}}{\norm{\hat{y_j}}}\Bigg>^2\Bigg)$ (6)
I am having trouble deriving (6) from (5). Can you show me how to derive this?
$\norm{\dfrac{\partial \hat{L}}{\partial y_{j}}}^2 = \Big( \dfrac{\gamma^2}{\sigma_j^2} \Big)$$\Bigg( \norm{\dfrac{\partial \hat{L}}{\partial z_{j}}}^2-\dfrac{1}{m}\Bigg< 1,\dfrac{\partial \hat{L}}{\partial z_{j}} \Bigg>^2 - \dfrac{1}{m}\Bigg<\dfrac{\partial \hat{L}}{\partial z_{j}},\hat{y_j} \Bigg>^2 \Bigg)$ (7)
Thank you.


texts: Is it acceptable to use label encoding for nominal categorical data when one hot encoding would create too many features?

texts:
I'm working on a short data science project to compare the accuracy of different classification methods. The groups decided to use and compare Random Forest, Naive Bayes and SVM.
The dataset we are using has four categorical features. Each of which has a large number of unique values.

There are 16537 unique combinations of 17370 unique values in
FeaureA.
There are 13860 unique combinations of 13852 unique  values
in FeaureB.
There are 3295 unique combinations of 29 unique  values
in FeaureC.
There are 1518 unique combinations of 29 unique  values
in FeaureD.

From what I've read the RF and NB algorithms should work fine with label encoding but SVM requires one hot encoding. However that would increase the number of features by ~35K. The performance cost seems like it would be significant. Ideally we would use the same encoding for all three algorithms. Would it be better to take the performance hit and try something like PCA for feature reduction?


texts: Which machine (or deep) learning methods could suit my text classification problem?

texts:
I am long-time engineer with almost zero machine learning experience, who is trying to determine a good starting point to solve my problem (hopefully using machine learning).

The problem (I'll keep it simple):

Ultimately, I wish to be able to automatically assign a category to a financial transaction description
For example: "121217 POST XX123 TONYS COFFEE" with an amount of $5, should map to "Food & Drink"
Transaction descriptions are unstructured and often consist of unnatural language; also sometimes words can be truncated, or concatenated to others words
There may or may not be "features" such as country/region codes, dates, etc., in a description
Overall, there is no guarantee to the order or structure of the tokens in a description
A transaction will always have an amount

Assumptions:

I maintain a list of potential categories (maybe, 20 or 30 in total)
I could maintain a huge list of business names mapped to their distinctive category (may not be necessary to use, though?)
There is an existing set of training labelled data (raw descriptions/amounts and the category they belong to; in the thousands, not millions)

A best effort to extract a business name from a description, could be considered, but it would be great if that was not necessary for the accuracy of a final system.

I was originally thinking about NLP somewhat, but as this is fairly non-natural language, with no semantics, I believe there is no use of NLP. NER (named entity recognition) is maybe not really helpful either, as it is generally required to understand a text somewhat in order to determine entities.

I am toying with the idea of downloading GloVes pre-trained word vectors (https://nlp.stanford.edu/projects/glove/), to help determine words related to categories, though I am unsure how as of now or how well that might work. The idea might be if I trained something to say "Jimmy's Diner" -> "Food & Drink", then, for example, "Bobby's Rest" might map to "Food & Drink" too, as that is the nearest category to it in terms of word relationships/distance. It's depends on the possibility of being able to query the word embedding in such a way, as well as train it.

I guess in other to train a system using labelled data, I'd need to extract features from the description. The problem is, what features? Some features might be useless (unique identifiers, concatenated words, etc.). I'd need the system to be somewhat forgiving in terms of polluting it with useless features (avoidable to a degree, but probably not inevitable).

Either way, it would be great to hear how some of you experts might begin to approach this: what ML techniques would you deem most suitable?

I have researched machine learning and deep learning & the associated frameworks quite a bit over the last few days, but there is so many areas, with so much potential, that it is hard to to know where to begin.


texts: Accuracy keep changing by changing randomState of classifier

texts:
I try to classify car sound samples. Using MLPClassifier from Scikit. I'm getting very different and confusing test results between 2 different test sets, and I am stuck:

Training is done with the first data set of 1500 samples, splitted as 70/30 train/test. Second set is isolated, I use only for the final testing, 700 samples completely unseen data.
First test set is stable and always around %90 test, %99 train set accuracy.
Second test is completely random, it changes while I change the randomState variable of the classifier. It can be %20 or %80 accuracy.

This is extremely frustrating. The difference between two sets is first one is mainly from sounds that come with UrbanSound data collection. And second set is more real world, I recorded with an iPhone. I checked that they all have the same duration, sample rate, and bit rate.

So my question is:

if your accuracy changes randomly by changing your randomState of your classifier, on a certain test set, what does this tell you about your data?
And second what would be my approach now? Totally lost.

ps: My features are first 20 of the mfcc coefficients. OR 60 band of mel spectogram. I try different things.

I am also wondering if all these code examples, academic papers about sound classification which uses the sound samples from UrbanSounds and ESD50 sets, did they ever test their accuracy with completely random real world sounds, recorded and processed with different tools ?!

Below is when I plot these 2 different sound sets(only the positive class) with:

  plt.plot(car_features_1,'.')
  plt.plot(car_features_2,'.')

They look quite different to me, they are both car sounds one is recorded by iphone other is coming from Urbansounds .etc

This is the code for 60 features(60 from mel spectogram ):

clf = MLPClassifier(activation='relu', solver='adam', alpha= 1, hidden_layer_sizes=(60, 60, 60), random_state=None, max_iter=2000)

clf.fit(X_train, y_train)

And this is how i extract features

 mel = np.mean(librosa.feature.melspectrogram(X, sr=sample_rate).T,axis=0) #128 array
 mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=20).T,axis=0) #20 array


texts: Can I expect necessarily higher accuracy when feature selection/dimension reduction is used to select a subset of features?

texts: The feature selection/dimension reduction is performed to eliminate irrelevant or redundant features so it will improve the computation efficiency (less computationally expensive). My question is that can we expect any changes in the accuracy of prediction when efficient features are used for classification vs when all features used?


texts: Dealing with categorical variables in regression problems which method to use?

texts:
Usually if I have regression problem and my initial dataset contains categorical variables like :
column 1:
Math
Science
Science
English

I would convert this non-numerical variables to numerical variable such that : Math: 0, Science : 1, English : 2.
However, I recently found a tutorial said that this solution is not performant because there is no favorite class among other means there is no increase between those classes and if it existe we can not quantify it.
Can anyone explain this for me because I usually worked with solution one ?


texts: How do subsequent convolution layers work?

texts:
This question boils down to "how do convolution layers exactly work.

Suppose I have an $n \times m$ greyscale image. So the image has one channel.
In the first layer, I apply a $3\times 3$ convolution with $k_1$ filters and padding. Then I have another convolution layer with $5 \times 5$ convolutions and $k_2$ filters. How many feature maps do I have?

Type 1 convolution

The first layer gets executed. After that, I have $k_1$ feature maps (one for each filter). Each of those has the size $n \times m$. Every single pixel was created by taking $3 \cdot 3 = 9$ pixels from the padded input image.

Then the second layer gets applied. Every single filter gets applied separately to each of the feature maps. This results in $k_2$ feature maps for every of the $k_1$ feature maps. So there are $k_1 \times k_2$ feature maps after the second layer. Every single pixel of each of the new feature maps got created by taking $5 \cdot 5 = 25$ "pixels" of the padded feature map from before.

The system has to learn $k_1 \cdot 3 \cdot 3 + k_2 \cdot 5 \cdot 5$ parameters.

Type 2.1 convolution

Like before: The first layer gets executed. After that, I have $k_1$ feature maps (one for each filter). Each of those has the size $n \times m$. Every single pixel was created by taking $3 \cdot 3 = 9$ pixels from the padded input image.

Unlike before: Then the second layer gets applied. Every single filter gets applied to the same region, but all feature maps from before. This results in $k_2$ feature maps in total after the second layer got executed. Every single pixel of each of the new feature maps got created by taking $k_2 \cdot 5 \cdot 5 = 25 \cdot k_2$ "pixels" of the padded feature maps from before.

The system has to learn $k_1 \cdot 3 \cdot 3 + k_2 \cdot 5 \cdot 5$ parameters.

Type 2.2 convolution

Like above, but instead of having $5 \cdot 5 = 25$ parameters per filter which have to be learned and get simply copied for the other input feature maps, you have $k_1 \cdot 3 \cdot 3 + k_2 \cdot k_1 \cdot 5 \cdot 5$ paramters which have to be learned.

Question

Is type 1 or type 2 typically used?
Which type is used in Alexnet?
Which type is used in GoogLeNet?

If you say type 1: Why do $1 \times 1$ convolutions make any sense? Don't they only multiply the data with a constant?
If you say type 2: Please explain the quadratic cost ("For example, in a deep vision network, if two convolutional layers are chained,
any uniform increase in the number of their filters results in a quadratic increase of computation")

For all answers, please give some evidence (papers, textbooks, documentation of frameworks) that your answer is correct.

Bonus question 1

Is the pooling applied always only per feature map or is it also done over multiple feature maps?

Bonus question 2

I'm relatively sure that type 1 is correct and I got something wrong with the GoogLe paper. But there a 3D convolutions, too. Lets say you have 1337 feature maps of size $42 \times 314$ and you apply a $3 \times 4 \times 5$ filter. How do you slide the filter over the feature maps? (Left to right, top to bottom, first feature map to last feature map?) Does it matter as long as you do it consistantly?

My research

I've read the two papers from above, but I'm still not sure what is used.
I've read the lasagne documentation
I've read the theano documentation
I've read the answers on Understanding convolutional neural networks (without following all links)
I've read Convolutional Neural Networks (LeNet). Especially figure 1 makes me relatively sure that Type 2.1 is the right one. This would also fit to the "quadratic cost" comment in GoogLe Net and to some practical experience I had with Caffee.


texts: Probabilistic Outlier Detection (edited + clarified)

texts:
Measured data $D \in \mathbb{R}^3$, every $d^i \in D$ is $d^i_{(x)}$, where the  $x=[x_1, x_2]$. Simply said, the measured data are function of $x$. It is known, the dependency is linear, such as:

$$d^i_1 = a_1^Tx+b_1$$
$$d^i_2 = a_2^Tx+b_2$$
$$d^i_3 = a_3^Tx+b_3$$

where $a_{1-3} \in \mathbb{R}^2$ and $b_{1-3} \in \mathbb{R}$, so that $d^i_{1-3}$ is a set of 3 simple 2D planes, over $x=[x_1, x_2]$.

While it is known, that $d^i_{(x)}$ is function of $x$, the $x$ is not measured. Therefore the parameters $a_{1-3}$ and $b_{1-3}$ are difficult to find. The data interpretation is simplified to 3 dimensions for visual clarity, real problem is of higher dimension.

Data Example

The figures below show $d^i_{1-3}$, as a function of $x=[x_1, x_2]$, assuming the $x$ is measurable. This is only to visualize the data set.

dimension 1 of $d^i$

dimension 2 of $d^i$

dimension 3 of $d^i$

Problem formulation

How to assign probability $p(d^i)$ to measured $d^i$, such that it reflects the fact that it lies in the feasibility region (the 3 green hyper-planes) ?

1) Simply said, how to train the model using $d^i = [d^i_1, d^i_2, d^i_3]$ to spot the outlier $d' = [d^q_1, d^q_2, d^k_3]$. The outlier is marked by red dot and deviates only in the 3rd dimension. (please, suggest ways that work in higher dimensions as well).

In practice, $x=[x_1, x_2]$ is not measured !

2) Does the knowledge of the linearity of $d^i_{(x)}$, help ? What if the dependency is not linear ?

Idea

I think I can have 3 simple Gaussian PDFs, where each PDF gives the probability for one {$d^i_1$, $d^i_2$, $d^i_3$}, and the parameters (mean and variance) are function of the other 2 $d^i_{1-3}$. For example:

$$p(d^i_1) = N(d^i_1|\mathrm{E}_{(d^i_2, d^i_3)},\mathrm{Var}_{(d^i_2, d^i_3)})$$

then it follows for the other 2 dimensions:

$$p(d^i_2) = N(d^i_2|\mathrm{E}_{(d^i_4, d^i_3)},\mathrm{Var}_{(d^i_4, d^i_3)})$$
$$p(d^i_3) = N(d^i_3|\mathrm{E}_{(d^i_1, d^i_2)},\mathrm{Var}_{(d^i_1, d^i_2)})$$

P.S.

The outlier $d'$ is placed at the border (y=0), just to enhance the visibility. The outlier can be measured with any value of $x=[x_1, x_2]$.


texts: How does training a ConvNet with huge number of parameters on a smaller number of images work?

texts:
I have two questions:

I am wondering why is that a very deep model such as VGG-16 which has approximately 138 million parameters (Source) can be used as a model to be trained on just 1.3 million images? Even though the authors of VGG-16 used dropout and regularisation to avoid overfitting (source), I just feel that the amount of parameters is still significantly more than the number of images.
Is it also true that when you use $N$ number of epochs, you "effectively" have $K*N$ training datasets, where $K$ is the number of training datasets you have? In the case above, $K$ = 1.3 million. Personally I don't think this is the case, but I am not sure.

I probably have not done research or read thoroughly, so I would like to apologize if the answers are already somewhere out there. Thank you.


texts:
I want to predict the demand for a product through time series methods like ARIMA. In addition to historical demand data, I have the data of factors affecting the demand, that is, contextual data. How would it be possible to combine the historical data with contextual data in time series?

I would be thankful if it is possible to share some references. Moreover, I am willing to use Python for prediction. Is there any library in Python for this purpose?


texts:
I am working on this course on Machine Learning 2012 from UBC (CPSC 340). I am stuck on a Homework code problem which shows the following RuntimeError in RuntimeWarning: invalid value encountered in add, C = np.diag(np.dot(X.T, (ytile.T - protile.T + third.T)))

Can anyone help me check what the problem in the code is? Also, when I print the $\theta$, they are exponentially increasing and the loop keeps on executing.

def ridge(X, y, d2):
    one = np.linalg.inv(np.dot(X.T, X) + d2 * (np.eye(8)))
    two = np.dot(X.T, y)
    theta = np.dot(one, two)
    return theta

def lasso(X, y, d2):
    theta = ridge(X, y, d2)
    k = len(theta)
    theta_ = np.zeros(k)
    A = 2 * (np.sum(X**2, axis = 0))
    while np.sum(np.abs(theta_ - theta)) > 1e-5:

        theta_ = theta.copy()

        ytile = np.tile(y, (k, 1))
        pro = np.dot(X, theta)
        protile = np.tile(pro, (k,1))
        third = np.dot(np.diag(theta), X.T)
        ########################################################
        C = np.diag(np.dot(X.T, (ytile.T - protile.T + third.T)))
        ########################################################
        for i in range(k):
            if C[i] < -d2 :
                theta[i] = ( C[i] + d2 ) / A[i]
            elif C[i] > d2 :
                theta[i] = ( C[i] - d2 ) / A[i]
            else:
                theta[i] = 0
    return theta

theta = lasso(Xtrain, ytrain, 1)
print(theta)


texts: Why do we use the F1 score instead of mutual information?

texts:
We often use the classification threshold that maximizes the F1 score, if we don't have a prior cost function of false positives and false negatives.

This balances the desire for precision and recall. If either one is 0 the F1 score is 0; and if we have a perfect classification the F1 score is 1.

On the other hand I'm hard pressed to find a scientific justification to maximize F1 in general, or a business problem where F1 is the thing we need to maximize.

F1 is not symmetric. If we have an 60/40 binary distribution and choose the 40% class as the positive class, and we classify everything as positive, we get 100% recall and 40% precision for F1 score of 0.4. (F1=0 if we classify everything negative). If we choose the 60% class as positive, and classify everything positive, we get F1 score of 0.6.

Why not use mutual information, which minimizes the surprise in the prediction vs. actual?

When we estimate probabilities by minimizing log loss, we are also minimizing K-L divergence or entropy or surprise in the information theory sense. If a 50/50 probability prediction contains 1 bit of entropy or surprise, minimizing the log loss minimizes the number of bits of entropy or surprise in our predictions vs. actual.

And in information theory, if we don't have probabilities and want to measure the information transmitted by a noisy signal, we use mutual information.

And mutual information is symmetric, and 0 for a signal of all 1s or 0s. Predicting all 1s or 0s in some sense gives no information about the ground truth of the response variable.

In a way, a prediction is like a noisy channel from the present to the future, and an information theory concept like mutual information seems like a well-grounded criterion to choose the classification threshold, and F1 seems arbitrary.

Is there any reason why F1 is preferred over mutual information?


texts: Why are observation probabilities modelled as Gaussian distributions in HMM?

texts:
HMM is a statistical model with unobserved (i.e. hidden) states used for recognition algorithms (speech, handwriting, gesture, ...). What distinguishes DHMM form CHMM is the transition probability matrix P with elements. In CHMM, state space of hidden variable is discrete and observation probabilities are modelled as Gaussian distributions.

Why are observation probabilities modelled as Gaussian distributions
in CHMM?
Why are they (best)distributions for recognition systems
in HMM?


texts: CNN only performs well when split into 2 models

texts: I have 2 groups of data of equal shape (the main difference between the 2 are that one has half as many features - and consequently different labels of course)that perform better when they are trained independently (both using the same code). Is there any canonical approach to create a single model that performs well on both from here? Of course, I have some ideas such as increasing the number of layers and neurons but was wondering if there is something I am ignorant of. (p.s. this is for a regression problem)


texts: How to tackle a multilabel classification problem

texts:
I am trying to build a LSTM model for a multiclass classification problem on textual data. Until now, I have only built a model when one input belongs to one of the categories. What do I do when one input can belong to more than one class (i.e.: one entry of data can belong to 2-3 categories)? Can anyone help me with some blogs or resources to build an intuition for making such a model?

Thanks.


texts: Non-mutal exclusive classification task examples

texts:
I am reading the excellent Hands-on Machine Learning with Scikit-Learn and TensorFlow and in chapter 10, the author says:

  "For the output layer, the softmax activation function is generally a
  good choice for classification tasks (when the classes are mutually
  exclusive). For regression tasks, you can simply use no activation
  function at all."

All classification problems I can think of (binary classification, image classification, etc) generate mutually exclusive classes.

Can someone give me a few examples of non-mutual exclusive classification problems?


texts: Converting similarity matrix before inputting to t-sne

texts:
I have a cosine similarity matrix where I want to adjust it to inputto t-sne. I fond the following explanation in a FAQ. As mentioned there I have made the diagonals to zero. what does it mean by symmetrize the pairwise similarity matrix, and normalize it to sum up to one?

  Can I use a pairwise similarity matrix as input into t-SNE?

  Yes you can! For instance, we successfully applied t-SNE on a dataset
  of word association data. Download the Matlab implementation, make
  sure the diagonal of the pairwise similarity matrix contains only
  zeros, symmetrize the pairwise similarity matrix, and normalize it to
  sum up to one. You can now use the result as input into the tsne_p.m
  function.


texts:
I've a trainingset which has 400 features and most of them have null value.

I tried to draw the heatmap of nullity correlation matrix by means of Python and missingno, but the heatmap is unreadable due to high number of features.

How can I print the nullity correlation matrix, instead of draw it?


texts: Dynamic pricing with aggregate constraints

texts:
so I have this situation. I am trying to understand whether my customers will buy my product at a certain price based on a previous offer made to them. Specifically, I have a lot of data on my clients and I have rolled a previous "campaign" round and I saw who bought it and who did not. Now I am trying to calibrate new prices for those who did not and I wanted to find prices that maximize the chances of them buying it but at the same time I have an aggregate constraint that I have to uphold (a weighted aggregate profit margin).

I am in the early stages but I wanted to understand the models of reference for this kind of problems. Any suggestion to which area/material is most relevant is very appreciated.

p.s.: I have tried to scour the forum for similar questions without success but, in case I missed something, do close the question and kindly refer me to that content.


texts: How can I implement my own AR(p) simulation algorithm?

texts:
Recall that an AR(p) process can be defined as
$$
X_t = \phi_1 X_{t-1} + \cdots + \phi_p X_{t-p} + Z_t
$$
where $Z_t$ is I.I.D. white noise. I want to write an algorithm simulating these processes, but I'm not sure if I'm taking the right approach. Here's my idea:

INPUT:
  n = length of time series
  [\phi_1,...,\phi_p] the parameters of the process

1. Construct a white noise vector Z of length n + p
2. Have the first rolling sum of the X_i's be set as the noise
3. Construct the rest of the time series from there

In this case
$$
\begin{align}
X_0 &= \phi_1Z_{-1} + \cdots + \phi_pZ_{-p} + Z_0 \\
X_1 &= \phi_1X_0 + \phi_2Z_{-1} + \cdots + \phi_pZ_{-(p-1)} \\
&\cdots
\end{align}
$$

Another option would be to start by constructing $X_1,\ldots, X_p$ by having the lower order terms in the recursion equation be set to 0. So, for the first few
$$
\begin{align}
X_1 &= Z_1 \\
X_2 &= \phi_1 X_1 + Z_2 \\
X_3 &= \phi_1 X_2 + \phi_2 X_1 + Z_3 \\
&\cdots
\end{align}
$$
Which approach is the "right" approach for constructing the simulation?


texts: Can we use CNN to effectively learn from a table whose various rows are permuted in the testing dataset, output for each remains same?

texts:
I have a set of classes, 37 to be precise. Each class is represented by a feature vector of size [1,10].
A single input sample has the dimensions of [6,10], where each row in this sample represents a different class.
Input only has one arrangement of these rows, whereas during testing, the rows can be permuted in the input sample and output should remain the same.
Can I use CNN to drop the need of training on every permutation by using a kernel of size [1,10] which will effectively run slide vertically on the input sample?
The size of the training dataset is considerably less(500 samples).


texts: Returning rows from a list of indexes in Python Pandas

texts:
Context
I have a CSV containing two types of rows, an observation record and on the row below, an observation value which is related to the observation record above. The record line contains a four-letter code which denotes the type of observation. My goal is to create a new CSV containing only those observation records which match a particular list of codes, along with the associated observation values from the row below.
Example from file
OBSERV\LTRC,CL1,0,10.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,10,20.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,20,30.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,30,40.00;
OBVAL\14,,0.5,V;

Code so far
import pandas as pd
data = pd.read_csv(r"CSVFILEPATH")
df = pd.DataFrame(data)
df.set_index
newdf = df.loc[df[0].str.contains('LLRT|LLTX|LRRT|LTRC|LV10|LV3|LEDR|LTRV|LES2|LES1', regex = True)]
# this returns all observation record rows I care about but I still need the associated observation values.

keep_ind = [] #This list will contain all indexes to keep

observ_ind = ndf.index.values.tolist()#The list of observation record indexes to keep
keep_ind.append(observ_ind)#Added these to the keep_ind list

The Question
How do I take this list of indexes (keep_ind), append a new list which is the same list with 1 added to each item (to get all of the observation value rows beneath the records) and create a new dataframe which contains all of the rows at each of these indexes in this combined list?
So far, I've tried:
keep_ind.append(observ_ind + 1 for i in observ_ind)

But this gives the error:
generator object <genexpr> at 0x0000028057C33648>]


texts: Fitting a model as a significance testing alternative

texts:
My colleagues use somewhat unusual approach to estimate how far away performance of some of our company related processes diverge from historical levels. Their nature isn't really relevant for the purpose of this discussion. We may just assume here, that the final result can be represented with some real number which fluctuates over time and has some seasonality components.

They fit a regression model (with XGBoost) on historical data excluding the most recent period (say, the last month) and than compare the most recent values with model's forecast.

I argue, that this is basically an overkill and some sort of a crooked statistical significance test. And rather than fitting a model and making a forecast, we should actually calculate some normalised historical indicators (stratified by relevant groups and taking into account seasonality, where appropriate) and then do a standard significance test using a significance level, that would make sense from a business perspective.

Am I right or not? Are there any better ways to approach a problem of estimating performance and comparing it to historical data?
Does approach with modelling/forecasting has any legs? How it can be justified from a statistical standpoint?


texts: I have limited samples for one class, unlimited samples for the other class. Need to balance?

texts:
I want my machine learning algorithm to learn the difference between two classes, actually picture of X or picture of something else.

My sample data is:

500 pictures of X (I know it is low, unfortunately I can't do much about it)
1,000,000 pictures of something else

Question: Should I do the training with all of the 1,000,000 pictures of something else?
Or would such an imbalance have negative effects? For instance would it kind of "drown" the other data?

Notes:

Computing power and time is not an issue.
In the real world, pictures of X make up 5% to 10% of the data, so I don't think I have a class imbalance problem.
I think the classification is an easy one, machine learning will probably be able to understand quickly.
I am OK with a reasonable amount of misclassifications.
In case that matters, I am planning to use Keras and Tensorflow with Flatten/Dense relu/Dense softmax/AdamOptimizer/sparse_categorical_crossentropy.


texts: Why the validation accuracy does not increase in a normal way over the epochs?

texts:
I'm trying to transfer learning VGG16 model with imagenet in a dataset of retinal images but i'm confused to get a graph like this in the picture below, I don't know why the validation accuracy didn't increase in a normal way over the epochs, like training accuracy did, is it an index of overfitting ? if yes, how can i overcome it ?

STEP_SIZE_TRAIN = len(train_generator)
STEP_SIZE_TEST = len(test_generator)

Network = ModelDefinition(base_model)

# don't train existing weights
for layer in Network.layers:
    layer.trainable = False
# Number of classes
folders = glob(train_path + '/*')
# our layers
x = Flatten()(Network.output)
# a layer instance is callable on a tensor, and returns a tensor
prediction = Dense(len(folders), activation='softmax')(x)

# This creates a model that includes
# the Input layer and One Dense layers
model = Model(inputs=Network.input, outputs=prediction)
#view the structure of the model
model.summary()
#tell the model what cost and optimization method to use
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

r = model.fit_generator(train_generator, steps_per_epoch=STEP_SIZE_TRAIN, validation_data=test_generator,
                        validation_steps=STEP_SIZE_TEST, epochs=40)

Model_fileName = 'KERAS/MODELS/all-D0-D14/' + base_model
train_datagen = ImageDataGenerator(rescale=1. / 255, shear_range=0.2,
zoom_range=0.2, horizontal_flip=True)
train_generator = train_datagen.flow_from_directory(train_path, target_size=
(224, 224), batch_size=32,class_mode='categorical')

test_datagen = ImageDataGenerator(rescale=1. / 255)
val_generator = test_datagen.flow_from_directory(test_path, target_size=(224,
224), batch_size=32,class_mode='categorical', shuffle=True, seed=42)


texts: can I use z-score normalization even if it doesn't make sense for my data to be negative?

texts:
I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for it to have negative values. Let's say for example speed or distance, it doesn't make sense that speed would have a negative value after normalization! is it logical to think like this or am I wrong and it is perfectly fine to use z-score even if the data will be negative and it doesn't make sense?

ps: I know velocity can be negative if we are talking about Vectors but I meant to say that I have a discrete Values for Speed or Distance a.k.a Length of something which cannot be negative.


texts: Proximity multi-dimentional arrays. Which algorithms are commonly used?

texts:
I'm new to data science so still learning so much. I've been searching for proximity algorithms but I'm not sure which are suited for multi-dimensional arrays. Any guidance would be greatly appreciated.
Here are the arrays I'd like to determine of how similar/dissimilar they are:
Entity 1: (base comparison)
| Chair | Table | Cup |
|  2    |   3   |  2  |

Entity 2:
| Chair | Table | Spoon |
|  2    |   3   |   1   |

Entity 3:
| Chair | Table | Door |
|  3    |   3   |  1   |

Entity 4:
| Chair | Room | Door |
|  2    |   3  |  1   |

In this example Entity 1 is close to Entity 2 because they both have a Chair with a value of 2 and a Table with a value of 3. The next best match for Entity 1 is Entity 3 because the Table match (again) is identical but the Chair is 3 which is only one point higher than 2.
The matching is based on the object first (Chair, Table, Cup, Spoon, Door) and then the number closest to the base Entity. So if we're comparing [Chair, 2] then [Chair, 1] and [Chair, 3] would have the same proximity since 1 and 3 are only 1 point off from 2. However [Chair, 0] or [Chair, 4] would be less of a match because they are 2 points away. And of course [Spoon, 2] is not at all similar to [Chair, 2] because we care about the object type.
Sorry if this seems trivial, I'm still learning.
Thank you for any guidance.


texts:
Which treebanks are based on an XML format?
What is the advantage of XML format for a treebank? I think it may have effects on annotation and querying the treebank.

for example LASSY and Alpino or TIGER are in xml format.


texts: Is it good practice to use SMOTE when you have a data set that has imbalanced classes when using BERT model for text classification?

texts: I had a question related to SMOTE. If you have a data set that is imbalanced, is it correct to use SMOTE when you are using BERT? I believe I read somewhere that you do not need to do this since BERT take this into account, but I'm unable to find the article where I read that. Either from your own research or experience, would you say that oversampling using SMOTE (or some other algorithm) is useful when classifying using a BERT model? Or would it be redundant/unnecessary?


texts: Transform Categorical Variables into Numerical

texts:
I'm very new to machine learning approaches. I'm reading a tutorial for build a predictive model using random forests.
One of the transformations implemented was transform categorical variables to binary.
Imagine (short sample):

Field_Desc  Field_Value
A               32
A               100
B               1

And then the developer pass this dataset into:

Field_A1    Field_B1    Field_Value
1               0                32
1               0                100
0               1                 1

What is the advantage to make this transformation for Random Forest Prediction? And for K-Means there will have any advantage?

Thanks!


texts: Replace column in a dataframe with another column based on index

texts:
I have the following dataframe A:
        score   content  content_preprocessed   for_analysis
1         5       a             aaa                 True
2         5       b             NaN                 False
3         1       c             ccc                 True
4         1       d             ddd                 True

I need to replace content_preprocessed column with the following new column from dataframe B:
        content_preprocessed
1              aaz
3              ccf
4              ddo

which contains new values for content_preprocessed where for_analysis is set to True. The values for content_preprocessed should remain NaN if the corresponding for_analysis is False.
How do I do that?


texts: Different scales for 2D plots grouped in 3D perspective in python

texts:
I'm trying to realize a plot similar to the one in the following figure:

I started from the code contained in the answer here:
https://stackoverflow.com/questions/34099518/plotting-a-series-of-2d-plots-projected-in-3d-in-a-perspectival-way

m3dplot and 3D collections seems to work well, however, I have problem with including axes with different heights in the same plot. In the figure they should be different, but in the example attached and in the documentation I found only plots with only one scale. Is there a way to achieve that using matplotlib?

At the moment this is the rough outcome that I obtained:

This is a MWE to get an idea:

import matplotlib.pylab as pl
from mpl_toolkits.mplot3d import Axes3D
import numpy as np

x   = np.linspace(-2,2,100)
y1  = np.ones(x.size)
y2  = np.ones(x.size)*2
y3  = np.ones(x.size)*3
z1  = np.sin(x*10)
z2   = 10*np.sin(x*10)
z3   = 100*np.sin(x*10)

pl.figure()
ax = pl.subplot(projection='3d')
ax.plot(x, y1, z1, color='r')
ax.plot(x, y2, z2, color='g')
ax.plot(x, y3, z3, color='b')

ax.add_collection3d(pl.fill_between(x, 0.95*z1, 1.05*z1, color='r', alpha=0.3), zs=1, zdir='y')
ax.add_collection3d(pl.fill_between(x, 0.90*z2, 1.10*z2, color='g', alpha=0.3), zs=2, zdir='y')
ax.add_collection3d(pl.fill_between(x, 0.85*z3, 1.15*z3, color='b', alpha=0.3), zs=3, zdir='y')


texts: How to get a feature from sound/audio data learning using machine learning supervised classification?

texts:
I'm a bit confused about how data in audio files can be processed for a ML classification model. I have several .wav files that contain dogs barking and cats "meowing". The pipeline is the following:

Load the data
Convert the data to FFT for desired window
Apply MFCC filtering
Back-transform using DCT
Creating a "spectrogram" for the window
Training a model?

What I don't understand is:

if I have .wav files of different sizes, lets say 1 second and 0.8 seconds, I will end up with a different number of windows, if window size is 0.1 seconds then the first file will have 10 windows and the second will have 8 windows, so how can I feed this information to the learning algorithm consistently.
Does the algorithm learn from the entire .wav file or window by window?
If the algorithm learns from each window, will each window have a different prediction value?

thank you.


texts: Why ROC value area under curve of two models is different whereas accuracy, precision, recall, f1-score and confusion matrix is same

texts:
I am applying logistic regression and support vector machines on the extacly same dataset with 70% data for training and 30% for test. Both perform exactly the same have the same precision, recall, f1-score and confusion matrix. The confusion matrix and classification report for both can be seen below.

       [1180    0]
       [  17    0]]

         precision    recall  f1-score   support

   0       0.99      1.00      0.99      1180
   1       0.00      0.00      0.00        17

   accuracy                               0.99      1197
   macro avg          0.49      0.50      0.50      1197
   weighted avg       0.97      0.99      0.98      1197

I applied 10-fold cross validation to both models too both achieve the same mean accracy of 98.57% both failing to achieve any True Positives across all 10 folds, also have same number of TP,TN,FP,FN in each of the 10-fold

However, the problem is when I compute the roc area under curve values they are different as seen below:

Logistic Regression: 0.86
Support Vector Machines: 0.23

These values indicate logistic regression is much better as compared to support vector machines although there confusion matrix and classification report is the same I am really confused as all the other reuslts indicated same performance by both models ?? Can someone please help why both get different values of roc whereas everything else is the same.


texts: sk-learn - ValueError: array is too big.

texts:
I have a large dataset with characters and 90000 intances and I have the error ValueError: array is too big when I have the following code before the plot_kmeans_digits.py code:

data2=list(csv.DictReader(open('C:\diabeticdata.csv', 'rU')))
vec = DictVectorizer()
data = vec.fit_transform(data2).toarray()

Do you know how I can solve this error?

Thanks in advance.


texts: Is my bayes classification right or meaningful?

texts:
I have this dataset and I am learning about Bayes Classifier. After data cleaning, I have tried to use bayes classifier on it. I used R with this code:

library(klaR)
trainingChoco <- chocolateApriori[1:707,]
testChoco <- chocolateApriori[708:884,]
naivebayesChocolate <- NaiveBayes(RatFactor ~ ., data=trainingChoco)
predictionChoco <- predict(naivebayesChocolate, testChoco)
predictionChoco$posterior
library(caret)
tableChocolate <- table(predictionChoco$class, testChoco$RatFactor)
confusionMatrix(tableChocolate)

Where RatFactor only has 1, 2, 3, 4, 5 as values (I rounded the ratings). This is my result:

  predictionChoco$posterior
                  1            2           3           4            5

1434 5.987980e-14 2.619121e-05 0.080559640 0.919414168 5.987980e-11

1435 4.205489e-10 8.759363e-03 0.022926453 0.968314183 4.205489e-10

1436 4.205489e-10 8.759363e-03 0.022926453 0.968314183 4.205489e-10

1439 1.004950e-10 4.709587e-03 0.006445339 0.988845074 1.004950e-10

1442 9.257687e-10 3.133371e-04 0.004947920 0.994738741 9.257687e-10

1443 3.260598e-13 8.227926e-03 0.171718690 0.820053058 3.260598e-07

[...]

  confusionMatrix(tableChocolate)

Confusion Matrix and Statistics

 1  2  3  4  5

1  0  0  0  0  0

2  0  2  5  7  0

3  0  7 42 28  0

4  0  5 45 36  0

5  0  0  0  0  0

Overall Statistics

           Accuracy : 0.452

             95% CI : (0.3772, 0.5284)

No Information Rate : 0.5198

P-Value [Acc > NIR] : 0.97

              Kappa : 0.0431

Mcnemar's Test P-Value : NA

Statistics by Class:

                 Class: 1 Class: 2 Class: 3 Class: 4 Class: 5

Sensitivity                NA   0.1429   0.4565   0.5070       NA

Specificity                 1   0.9264   0.5882   0.5283        1

Pos Pred Value             NA   0.1429   0.5455   0.4186       NA

Neg Pred Value             NA   0.9264   0.5000   0.6154       NA

Prevalence                  0   0.0791   0.5198   0.4011        0

Detection Rate              0   0.0113   0.2373   0.2034        0

Detection Prevalence        0   0.0791   0.4350   0.4859        0

Balanced Accuracy          NA   0.5346   0.5224   0.5177       NA

Do you think this result is right? Or am I missing something? Can you explain what would you understand by looking at this result please? Or can you post a similar example done step-by-step? Thanks.


texts: What is Directional quantile envelopes and its application?

texts: According to this website, there are a lot of very cool graphs. But when I did research on "Directional quantile envelopes", there are minimum information about it. So I am wondering what exactly it is and how can I apply to data analysis?


texts: I need to write a SQL query where I need the data for the year 2014 from May-Dec and complete data (Jan-Dec) of the year 2015. Can anyone help me with this query?


texts: How to find lagged cross correlation between time series?

texts:
I have 2 time series, $X$ and $Y$, and I'm trying to find the best lag range that correlates $X$ to $Y$ (find the amount(s) of lag of $X$ that best correlate to the target variable $Y$).
For instance, if the best lag range is between $t = 8$ and $t = 10$, then the final equation would be $Y_t = \alpha_1 X_{t-8} + \alpha_2 X_{t-9} + \alpha_3 X_{t-10} + \alpha_4$.
Since the value of $Y$ could depend not only on some specific lag of $X$, but rather on a range of lags, I can't just find the correlation coefficient between $Y$ and different time lags of $X$ individually, and can't just run a regression model for several lags of $X$ as independent variables, since there are huge colinearities between those lags (series $X$ is a slow-changing time series).
Is there a way to find what are the best lags of $X$ to be used to predict my variable $Y$?


texts: Applying neural network for simple x^2 function for demonstration purpose

texts:
I have tried to train a neural network for a simple x^2 function

I developed training data in excel. First column (X) is =RANDBETWEEN(-5,5) i.e random integer between -5 and 5
Second column simply squares first column
And third column, my output 'y' column is 0 or 1. 0 if second column is less than 12.5 else 1

I made 850 training examples and used the first column as 'X' and third column as 'y'

However I am only able to get a training accuracy of 63%!

Where could I have gone wrong? I changed input_layer to 1 and tried hidden units between 5 and 35. Tried regularization lambda 0 to 2 but still only 63% accuracy! Where could I have gone wrong?

My predict function is p = 1 if h2(i)>0.5 else 0.

Any help will be much appreciated! :-)

I also noticed that my neural network's output is 0.3XXX for all training examples...how is this possible??


texts: How to combine two CART decision trees learned in same type of data?

texts:
We have distributed data centers and we build decision trees in each data center. Our problem is to combine our CART decision trees into one CART decision tree. The data in each data center related to the same event (data from light sensor for instance). I know about boosting methods but they don't give result we want to.

Is there any known method to do this ?


texts:
I am faced with a time series forecasting cold-start problem, specifically I am forecasting energy consumption of businesses where historic consumption data is available only for training but not new businesses. What I have for new (and existing) businesses is metadata such as business type, turnover etc.

I am trying to approach this problem as a multi-outcome regression where the predictors is represented by the metadata, and the outcomes are each time point in the consumption time-series. Thus, the outcomes are correlated. Some articles mentioned Predictive Clustering Trees (see this and this articles for example) as a possible solution to this problem.

Do you know if there is an implementation in Python or any similar algorithm? Alternatively, do you know of any other algorithm already implemented in Python that could be used for cold-start forecasting/multi-output regression?

Thank you.


texts: Which activation function of the output layer and which loss function are advised to be used for bounded regression?

texts:
I want my (deep) neural network to produce an output from a certain range, in my case between 0 and 255. I have scaled the labels from [0..255] to [0..1].

For the neural network, I have tried a couple of different architectures (number of layers, type of layers, etc.), and I've also tried with the following:

sigmoid activation function for the output layer with binary crossentropy loss
ReLU activation function for the output layer with MSE
some other combinations that don't necessarily make sense (out of desperation)

However, in every case so far, the network learns to saturate the output to the extremes of the values range, i.e. either 0 or 1 (even when I intentionally let the network try to overfit (by training for a long time on a very small dataset)).

I can provide further details, but my question is more general:

What is a go-to combination of activation function of the output layer and loss function to be used for getting network outputs from a full range of values that don't saturate in the extremes?

A similar question was already asked, but I didn't find the one answer there helpful.


texts: How to deal with features that have widely different dimensionalities

texts:
Say I'm building a neural network to fit some classifier of some sort. To make things concrete, let's take the example of predicting housing prices using features of houses.

What should I do if one or two of my features consist of many more numbers than the other features, or even all other features combined? For example, say I have a few housing features: size in sqft, age, median income of location. These are 3 numbers. And then I have another feature, height of the roof for each square foot of the house (it's a bit contrived for this example of course) for which I would have actually "size in sqft"-numbers for this feature. So now my feature vector looks like this:

X = [1500sqft, 34 years, $54,000, 10ft, 10.1ft, 10.3ft...1497 more numbers here...]

It seems that if I just naively put this into a neural net that the first 3 features would essentially be ignored since they only account for 3/1503 features. But they might actually be important. One try might be to simply average the "height of roof" feature over all of its elements to get an "average height of the roof" feature. That makes sense for this example, but what if sometimes I don't want to take this average?
Are there any industry practices on what I might try if I ran into a problem like this?


texts: How to do a "Model.deploy" inside a "PythonScriptStep" in Azure?

texts:
I have an AutoML Pipeline em a need to do an deploy models

train_step = AutoMLStep(name='AutoML_temporal',
    automl_config=automl_config,
    outputs=[metrics_data,model_data],
    enable_default_model_output=False,
    enable_default_metrics_output=False,
    allow_reuse=True)

extremidade_name = PipelineParameter("extremidade_name", default_value="extr001mgm")

deploy_step = PythonScriptStep(script_name="deploy.py",
                                       name="deploy",
                                       allow_reuse=False,
                                       arguments=["--extremidade_name", extremidade_name, "--model_path", model_data],
                                       inputs=[model_data],
                                       compute_target=compute_cluster,
                                       runconfig=aml_run_config)

I created the following script
#%% deploy.py
from azureml.core.model import Model, Dataset
from azureml.core.run import Run, _OfflineRun
from azureml.core import Workspace
from azureml.core.webservice import AksWebservice
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--extremidade_name", required=True)
parser.add_argument("--model_path", required=True)
args = parser.parse_args()

print(f"model_name : {args.extremidade_name}")
print(f"model_path: {args.model_path}")

run = Run.get_context()
ws = Workspace.from_config() if type(run) == _OfflineRun else run.experiment.workspace

from azureml.core.webservice import LocalWebservice, Webservice

deployment_config = AksWebservice.deploy_configuration(cpu_cores = 1, memory_gb = 1)
service = Model.deploy(workspace=ws, name = args.extremidade_name, models=[args.model_path])
service.wait_for_deployment(show_output = True)
print(service.state)

But I need to define inference_config
from azureml.core.environment import Environment
from azureml.core.model import InferenceConfig

env = Environment.get(workspace, "AzureML-Minimal").clone(env_name)

for pip_package in ["scikit-learn"]:
    env.python.conda_dependencies.add_pip_package(pip_package)

inference_config = InferenceConfig(entry_script='script.py',
                                    environment=env)

This is a continuous retraining scenario. How to read a 'path-to-score.py'? How to do I make this script?


texts: How to explain local minima found between two trained Neural networks?

texts:
I have trained 2 neural networks with SGD and then I have taken a linear path between their weights. Say W_0 and W_1 are the weight matrices of network 1 and network 2, respectively.

Then I compute W_new = (1-t)W_0+tW_1 for t ranging from -1 to 2, with small increments within this range. I then compute the loss for W_new at each t value. I get the following loss (it displays the cross-entropy and the accuracy), shown in the image below .

As you can see, the local minima of N_0 and N_1 happen at t=0 and t=1 as expected. However, there seem to be 2 local minima at t=-0.5 and t=0.5. I find this very strange and it has occurred over many pairs of neural networks.

This is a simple toy network with one hidden layer with 512 neurons and a relu activation.

Any idea as to why this is happening?